MCM Platform — 3-Tier Swarm Internal Reference & Architecture Guide

Confidential — For Internal Operations and DevOps Engineering Teams Only.
This document details the low-level architecture, multi-node clustering mechanics, Docker Compose compilation pipeline, internal security layers, scripting engines, and overlay network topography of the MCM (Multi-Cloud Management) Platform deployed on a 3-Tier Docker Swarm.

1. High-Level Orchestration & Topography

The 3-Tier production environment consists of 26 distinct containerized services distributed across a multi-node Swarm cluster consisting of one Swarm Manager (mcm-access) and two Swarm Workers (mcm-app and mcm-db).

2. Service Inventory

The table below lists all cluster-managed services, detailing their target node labels, internal networking ports, technology stack, and description.

3. Docker Compose Consolidation & Compilation Deep Dive

The installer does not deploy raw Docker Compose files. Instead, it consolidates four independent configuration sources into a single optimized deployment stack file (/opt/mcm/docker-compose-stack.yml).

3.1 The Merging Command

During installation or upgrade, the script runs the following command on VM1:

docker compose \
  --env-file /opt/mcm/internal_config.env \
  --env-file /etc/mcm/secrets.env \
  --env-file /etc/mcm/user_config.env \
  -f /opt/mcm/docker-compose-access.yml \
  -f /opt/mcm/docker-compose-app.yml \
  -f /opt/mcm/docker-compose-db.yml \
  -f /opt/mcm/docker-compose-swarm-override.yml \
  config | sed -E 's|image: /|image: |g' > /opt/mcm/docker-compose-stack.yml

Why this is done:

1. Modularity: Developers maintain clean service blocks split by tier (Access, App, DB).
2. Constraint Isolation: Node constraints and replica policies are kept in docker-compose-swarm-override.yml to prevent leakage into standard developer builds.
3. Secret Injection: Config attributes are pre-resolved from env files directly into the compiled output.

3.2 Compilation Transformations

Following the merge, the installer executes two critical post-processing steps to ensure Docker Swarm engine compatibility:

1. yq Sanitization

Docker Swarm ignores certain Compose parameters and throws schema validation errors if they remain in the file. The compiler strips them out using yq:

yq -i 'del(.services[].depends_on) | del(.services[].profiles) | del(.services[].links) | del(.name)' /opt/mcm/docker-compose-stack.yml

• depends_on is stripped because Swarm relies on dynamic service health checks and DNS availability rather than startup sequencing.
• profiles are stripped to force all services to compile into a single static stack representation.
• links are removed in favor of native overlay network service discovery.

2. Port Quote Stripping (Integer Resolution)

Docker Compose outputs port numbers wrapped in quotes (e.g. published: "80"), which causes Docker Swarm schema parser validations to fail (rejecting strings where integers are expected).

The compiler uses a Python regex replacement engine to remove quotes around these values:

import re
c = re.sub(r"published:\s*[\x22\x27](\d+)[\x22\x27]", r"published: \1", c)
c = re.sub(r"target:\s*[\x22\x27](\d+)[\x22\x27]", r"target: \1", c)

4. Node Placement Constraints & Replica Pinning

To enforce physical isolation, services declare placement constraints referencing labeled nodes.

During the install.sh sequence, nodes are labeled by their hostname:

• docker node update --label-add tier=access "$VM1_HOSTNAME"
• docker node update --label-add tier=app "$VM2_HOSTNAME"
• docker node update --label-add tier=db "$VM3_HOSTNAME"

Database Pinning & Data Volumes

Databases (postgres, mongodb, elasticsearch, wazuh-indexer) require persistent storage. In Docker Swarm:

1. Replica Constraints: All databases are pinned to replicas: 1.
2. Host Pinning: Databases are restricted to node.labels.tier == db. This forces Swarm to always schedule databases on VM3, binding their named persistent volumes (e.g., postgres_db_data, mongodb_data) to the local disk of VM3, preventing volume divergence across hosts.

5. Swarm Overlay Networking & Service Discovery

Services communicate across physical VM boundaries via the virtual apisix-net Overlay Network.

       VM 1 (Access Gateway)            VM 2 (Application API)           VM 3 (Data & Auth)
   +---------------------------+    +---------------------------+    +---------------------------+
   |   Ingress Traffic (443)   |    |                           |    |                           |
   |             |             |    |                           |    |                           |
   |          apisix           |    |       mcm-api (Java)      |    |          postgres         |
   +-------------+-------------+    +-------------+-------------+    +-------------+-------------+
                 |                                |                                |
                 +============= VXLAN Tunnel (UDP Port 4789) ======================+

5.1 Communication Protocols

1. Data Plane (VXLAN): Container-to-container packets are encapsulated and transmitted over physical host interfaces using UDP Port 4789.
2. Control Plane (Gossip): Cluster nodes exchange membership details and routing tables over TCP and UDP Port 7946 (configured as two separate rules).
3. Docker Swarm DNS: Docker runs an internal DNS resolver at 127.0.0.11 inside each container. Resolving a container name (e.g., mongodb or keycloak) returns a service Virtual IP (VIP). Swarm's built-in IPVS load balancer routes the request directly to the hosting container.

6. Multi-VM Installation and Upgrade Flows

6.1 Multi-Node Installation Flow (install.sh)

6.2 Multi-Node Upgrade Flow (upgrade.sh)

7. Certificate Architecture & Sync Deep Dive

MCM implements a dual Certificate Authority (CA) topology to secure both application-level traffic (via the MCM Root CA) and security monitoring telemetry (via the Wazuh CA):

7.1 Keystore and Truststore Internals

Java services require secure cryptographic storage formatted as PKCS12 archives:

• keystore.p12: Holds the individual service's private key, certificate, and CA chain.
  • Alias: mcm
  • Encryption Password: Read from KEYSTORE_PASSWORD in /etc/mcm/secrets.env
• truststore.p12: Holds trusted certificates representing entities the service can communicate with.
  • Alias mcm: Points to the MCM Root CA (ca.crt).
  • Alias wazuh-root-ca: Points to the Wazuh Root CA (root-ca.pem).
  • Aliases <domain>-<index>: Points to certificates fetched recursively from TRUSTED_DOMAINS (e.g. login.microsoftonline.com-0, login.microsoftonline.com-1).
  • Encryption Password: Read from TRUSTSTORE_PASSWORD in /etc/mcm/secrets.env

7.2 Configuration Fingerprint and Rotation

To prevent unnecessary certificate generation, the generate-certs.sh script tracks changes using /var/lib/mcm/.config_fingerprint.

The fingerprint is an MD5 hash calculated from:

Fingerprint = MD5(HOST_SERVER_IP | DOMAIN | TRUSTED_DOMAINS | MD5(secrets.env))

If the fingerprint changes or is missing:

1. The entire directory /var/lib/mcm/ is backed up to /var/backups/mcm/backup_<YYYYMMDD_HHMMSS>/.
2. A new MCM Root CA is generated at /var/lib/mcm/certs/ca.key and ca.crt.
3. Each microservice's keystore, truststore, and self-signed certificate are regenerated.
4. Wazuh certificates are regenerated using the generator container.
5. A new .config_fingerprint file is written.

7.3 Multi-Node Certificate Distribution Pipeline

• Generation: On execution of the installer, generate-certs.sh executes on VM1 (Access Node). It calculates the configuration fingerprint and generates CA files, keystores, and truststores inside /var/lib/mcm/ if changes are detected.
• Synchronization: The installer recursively copies /var/lib/mcm/ from VM1 to VM2 and VM3 over SSH prior to stack deployment, ensuring all cluster hosts share identical trust roots.

8. Wazuh SIEM Complete Reference

Wazuh operates as an independent security enclave using standard configurations and secure certificate authentication.

8.1 Deployment Profile: mcm-secops

Wazuh components are deployed dynamically under the mcm-secops Swarm profile:

• wazuh.manager: Central analysis engine performing file integrity monitoring (FIM), system auditing, and active response. Scheduled on VM3.
• wazuh.indexer: Security index database storing logs and vulnerability logs. Scheduled on VM3.
• wazuh.dashboard: Security console mapping alarms to compliance frameworks. Scheduled on VM3.
• init-wazuh-security: Configures certificate verification and index security mappings on first boot.

8.2 Certificate Generation & VM3 Distribution

Wazuh requires a strict directory structure of certificates under /var/lib/mcm/wazuh/certs/.

• Generation: The generate-certs.sh script runs wazuh-certs-generator in a container on VM1, which compiles and generates all Wazuh node certs (wazuh.indexer.pem, wazuh.manager.pem, wazuh.dashboard.pem, admin.pem) and writes them into .tar archives.
• Distribution & Extraction: The installer automatically transfers /var/lib/mcm/ over SSH from VM1 to VM3. It then extracts, flattens, and fixes file permissions to 755 under /var/lib/mcm/wazuh/certs/ on VM3 before Swarm services are launched.

9. Internal Configuration Reference (internal_config.env)

Location: /opt/mcm/internal_config.env (on VM1 - Access Node)

This configuration file is compiled during the packaging/release stage and contains registry locations, microservice image tags, and common environment configurations. In the 3-tier architecture, this file resides on VM1 and is sourced by the Swarm Manager during stack compilation.

10. Complete Variable & Secrets Reference

All environment settings, profiles, and backend credentials are split across user configurations and system-managed secrets.

10.1 User Configuration (/etc/mcm/user_config.env)

This file is created on VM1 (Access Node) and synced to VM2 and VM3 during installation. It controls the global deployment variables and active Swarm profiles:

10.2 Secrets (/etc/mcm/secrets.env)

This file contains randomly generated system passwords. These are generated via pwgen on first install:

11. Swarm Container Mounts & Volumes

Persistent data is mapped via named Docker volumes pinned to VM3 (the database node) to protect cluster storage integrity:

12. Dynamic Health Check Endpoints

Engineers can query local diagnostic checks directly on the hosting nodes: