MCMMCM DocsEngineering (Internal)
HLDModulesObservability
v1.2 is unreleased — see v1.1 for the current stable release.

Logs

Centralised log collection and search via Filebeat and Logstash across all supported resource types.

Logs

Filebeat runs on each monitored resource and tails log files and container output in real time. Raw events are forwarded to Logstash for parsing and enrichment, then indexed into Elasticsearch. The MCM UI provides a unified log viewer with full-text search across all resources.


Data Flow


Log Levels

LevelColourMeaning
ERRORRedApplication or system failure requiring immediate attention
WARNAmberDegraded state or approaching a threshold — may require action
INFOBlueNormal operational events (startup, deployment, health checks)
DEBUGMutedVerbose diagnostic output, typically disabled in production

Log Sources by Resource Type

Resource TypeLog Sources
AWS EC2/var/log/syslog, /var/log/auth.log, application log paths configured via filebeat.inputs
Ubuntu Host/var/log/syslog, /var/log/auth.log, application log paths — configured during onboarding
Azure VMLinux: /var/log/ · Windows: Windows Event Log via the Winlogbeat input
GCP Compute Engine/var/log/ + integration with GCP Ops Agent for structured log forwarding
Docker ContainerContainer stdout/stderr collected via the Docker socket (/var/run/docker.sock)
K8s Cluster / PodPod stdout/stderr via DaemonSet autodiscovery; control-plane logs: kube-apiserver, kube-scheduler, etcd

Logstash Pipeline

Logstash receives raw Filebeat events and applies the following processing stages before indexing:

  1. Grok parsing — extracts structured fields from syslog, nginx, and common log formats.
  2. JSON parsing — auto-detects JSON-formatted application logs and promotes fields to the document root.
  3. Metadata enrichment — appends account_id, resource_id, resource_type, and region from the Filebeat event tags.
  4. Geo-enrichment — resolves IP addresses in auth.log and HTTP access logs to country and city fields using the MaxMind GeoIP database.
  5. Timestamp normalisation — converts all log timestamps to UTC ISO 8601 and sets @timestamp.

Filebeat Configuration Reference

MCM writes the following base configuration. Additional filestream inputs are appended per application as needed:

filebeat.inputs:
  - type: filestream
    id: syslog
    paths:
      - /var/log/syslog
      - /var/log/auth.log

  - type: filestream
    id: app-logs
    paths:
      - /var/log/app/*.log

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

output.logstash:
  hosts: ["${LOGSTASH_HOST}:5044"]

For Kubernetes, Filebeat is deployed as a DaemonSet and uses the kubernetes autodiscovery provider to collect logs from all pods on each node automatically.

On this page