Observability Overview
Real-time metrics, log streaming, and distributed trace collection across cloud resources via the ELK stack.
Observability
The Observability module provides real-time infrastructure and application monitoring across three pillars — Metrics, Logs, and Traces. All telemetry is collected by Elastic Beats agents, enriched by Logstash, stored in Elasticsearch, and surfaced in the MCM UI.
Canonical Observability tool usage by provider and resource type is tracked in the Provider and Resource Reference.
Provider Support
Supported
AWS
Azure
GCP
Ubuntu
Docker
Kubernetes
Not Supported
GitHub- Docker Hub
Three Pillars
ELK Stack Architecture
Agent Deployment by Resource Type
| Resource Type | Agent Deployment | SSH Setup Required |
|---|---|---|
| AWS EC2 | MCM deploys Metricbeat + Filebeat + APM agent over SSH | Yes — via UI setup dialog |
| Ubuntu Host | Metricbeat + Filebeat installed during host onboarding | No — pre-configured |
| Azure VM | MCM deploys agents over SSH | Yes — via UI setup dialog |
| GCP Compute Engine | MCM deploys agents over SSH | Yes — via UI setup dialog |
| Docker Container | Metricbeat + Filebeat as sidecar containers via Docker labels | No — pre-configured |
| K8s Cluster / Pod | Metricbeat + Filebeat deployed as a DaemonSet in the cluster | No — pre-configured |
Resources that require SSH setup (EC2, Azure VM, GCP) display a Set Up Observability prompt in the MCM UI until the SSH credentials are provided. Resources that are pre-configured (Ubuntu, Docker, Kubernetes) skip this step entirely — Observability is active as soon as the resource is onboarded.
Use-Cases
- Real-time CPU and memory monitoring with configurable alert thresholds.
- Centralised log search across all cloud resources from a single UI.
- Distributed trace analysis to identify latency bottlenecks and error sources.
- Fleet-wide Observability dashboard showing resource health across all accounts.
- Per-account and per-resource drill-down into metrics, logs, and traces.
Tool Reference
| Tool | Role | Targets | Storage |
|---|---|---|---|
| Metricbeat | Metrics collection | All resource types | Elasticsearch |
| Filebeat | Log shipping | All resource types | Elasticsearch (via Logstash) |
| Logstash | Log parsing & enrichment | Log pipeline | Elasticsearch |
| Elastic APM Server | Trace ingestion | APM-instrumented applications | Elasticsearch |
| Elasticsearch | Central telemetry store | All three pillars | — |
Metricbeat
Targets: Linux/Windows hosts, containers, Kubernetes pods.
Modules used:
- System module: CPU, memory, network, disk I/O at configurable intervals (default 10s).
- Docker module: Container-level CPU, memory, network, and block I/O metrics.
- Kubernetes module: Node, pod, and namespace resource metrics via the Kubernetes API.
Filebeat
Targets: Log files, container stdout/stderr, Kubernetes pod logs.
Inputs:
filestreaminput for log files (/var/log/syslog,/var/log/auth.log, application log paths).containerinput for Docker container logs via the Docker socket.kubernetesautodiscovery for pod log collection in Kubernetes clusters.
All log events are forwarded to Logstash for parsing before landing in Elasticsearch.
Logstash
Role: Receives raw log events from Filebeat, applies grok patterns for syslog and common log formats, enriches events with resource metadata (account ID, region, resource type), and writes structured documents to Elasticsearch.
Elastic APM Server
Targets: Applications instrumented with Elastic APM agents or any OpenTelemetry SDK (via the OTLP endpoint).
Data collected: Trace ID, operation name, duration, HTTP status, span tree, and error details. See Traces for full field reference.