HLDModulesObservability
v1.2 is unreleased — see v1.1 for the current stable release.
Logs
Centralised log collection and search via Filebeat and Logstash across all supported resource types.
Logs
Filebeat runs on each monitored resource and tails log files and container output in real time. Raw events are forwarded to Logstash for parsing and enrichment, then indexed into Elasticsearch. The MCM UI provides a unified log viewer with full-text search across all resources.
Data Flow
Log Levels
| Level | Colour | Meaning |
|---|---|---|
| ERROR | Red | Application or system failure requiring immediate attention |
| WARN | Amber | Degraded state or approaching a threshold — may require action |
| INFO | Blue | Normal operational events (startup, deployment, health checks) |
| DEBUG | Muted | Verbose diagnostic output, typically disabled in production |
Log Sources by Resource Type
| Resource Type | Log Sources |
|---|---|
| AWS EC2 | /var/log/syslog, /var/log/auth.log, application log paths configured via filebeat.inputs |
| Ubuntu Host | /var/log/syslog, /var/log/auth.log, application log paths — configured during onboarding |
| Azure VM | Linux: /var/log/ · Windows: Windows Event Log via the Winlogbeat input |
| GCP Compute Engine | /var/log/ + integration with GCP Ops Agent for structured log forwarding |
| Docker Container | Container stdout/stderr collected via the Docker socket (/var/run/docker.sock) |
| K8s Cluster / Pod | Pod stdout/stderr via DaemonSet autodiscovery; control-plane logs: kube-apiserver, kube-scheduler, etcd |
Logstash Pipeline
Logstash receives raw Filebeat events and applies the following processing stages before indexing:
- Grok parsing — extracts structured fields from syslog, nginx, and common log formats.
- JSON parsing — auto-detects JSON-formatted application logs and promotes fields to the document root.
- Metadata enrichment — appends
account_id,resource_id,resource_type, andregionfrom the Filebeat event tags. - Geo-enrichment — resolves IP addresses in
auth.logand HTTP access logs to country and city fields using the MaxMind GeoIP database. - Timestamp normalisation — converts all log timestamps to UTC ISO 8601 and sets
@timestamp.
Filebeat Configuration Reference
MCM writes the following base configuration. Additional filestream inputs are appended per application as needed:
filebeat.inputs:
- type: filestream
id: syslog
paths:
- /var/log/syslog
- /var/log/auth.log
- type: filestream
id: app-logs
paths:
- /var/log/app/*.log
filebeat.autodiscover:
providers:
- type: docker
hints.enabled: true
output.logstash:
hosts: ["${LOGSTASH_HOST}:5044"]For Kubernetes, Filebeat is deployed as a DaemonSet and uses the kubernetes autodiscovery provider to collect logs from all pods on each node automatically.