Detailed architectural observations, scenario analysis, and deep dive into 1-Tier Monolithic vs 3-Tier Swarm performance dynamics.

Architectural Observations & Scenario Analysis

This document provides detailed scenario-by-scenario architectural observations and a deep-dive analysis into why the 3-Tier Swarm Cluster outperformed the 1-Tier Monolithic VM.

Architectural Observations & Scenario Analysis

The following section summarizes key architectural observations, performance behaviors, and resource bottlenecks identified across all 7 scenarios for v1.1:

Scenario 1: Baseline Multi-Module Performance (`enterprise-modules-test.js`)

For 1-Tier Monolithic: Every API iteration executes a fresh login operation, causing Keycloak PBKDF2/BCrypt password hashing to consume 451.3% CPU on the shared VM. This CPU saturation steals core processing cycles from microservices, pushing average response latency to 178.93 ms.
For 3-Tier Swarm Cluster: Offloading Keycloak to the dedicated DB Node (10.0.3.185) ensures microservice worker threads on the App Node (10.0.2.9) remain completely unthrottled, dropping average response latency by 2.5x down to 69.83 ms and increasing throughput to 103.34 req/sec.

Scenario 2: Peak Daily Stress Test (`peak-stress-test.js`)

For 1-Tier Monolithic: Reusing JWT tokens in the setup phase eliminates continuous login encryption overhead, dropping Keycloak CPU load from 451.3% down to 173.6% and freeing CPU capacity for MongoDB (512.2% avg CPU) and microservices. However, sharing a single host CPU bus under 100 VUs limits average latency to 375.16 ms.
For 3-Tier Swarm Cluster: Reusing JWT tokens combined with physical multi-node hardware isolation allows the App Node microservices to process 50,135 total requests at 274.09 req/sec (+74.4% higher throughput) with a 4.5x faster average latency of 83.76 ms (p95: 197.81 ms).

Scenario 3: Traffic Spike Tolerance (`spike-test.js`)

For 1-Tier Monolithic: A sudden 120 VU traffic surge creates CPU resource contention between MongoDB (506.9% avg CPU) and API services on the same host, resulting in 17 failed requests, a 422.79 ms average latency, and an elevated p95 latency of 1026.68 ms.
For 3-Tier Swarm Cluster: The cluster absorbs the rapid 120 VU surge with 0.00% HTTP errors across 51,427 requests. Microservices on the App Node recover to sub-38ms latency in < 15 seconds post-spike, achieving 253.94 req/sec throughput at 93.13 ms average latency (4.5x faster than 1-Tier).

Scenario 4: Sustained Endurance Test (`sustained-load-test.js`)

For 1-Tier Monolithic: Over a continuous 15-minute 50 VU endurance run with token reuse, the single VM operates reliably at 0.00% HTTP errors (77,991 requests processed), but shared host CPU queues result in an average latency of 78.21 ms (p95: 186.02 ms).
For 3-Tier Swarm Cluster: Sustaining the same 15-minute load on isolated Swarm nodes processes 78,043 requests at 0.00% HTTP errors while delivering 2.6x faster average latency (30.54 ms) and 3.2x faster p95 latency (58.15 ms) due to zero CPU queue contention on the App Node.

Scenario 5: Max Auth Capacity Test (`max-auth-capacity-test.js`)

For 1-Tier Monolithic: The single 16-vCPU VM allows Keycloak to burst up to 986.2% CPU (9.8 Cores) to execute 5,090 continuous logins without token reuse, completing all logins with 0.00% HTTP errors at an average latency of 178.03 ms.
For 3-Tier Swarm Cluster: Keycloak running on the 8-vCPU DB Node hits CPU saturation (445.8% peak CPU), causing 401/403 throttling (3.72% error rate at 100 VUs). However, successful requests achieve 2.3x faster average latency (77.30 ms) due to App Node separation.

Scenario 6: Max API Capacity Test (`max-api-capacity-test.js`)

For 1-Tier Monolithic: Ramping to 300 VUs causes severe CPU and disk I/O saturation (MongoDB at 475.0% avg CPU, FinOps API at 561.4% peak CPU), driving average latency to 1,632.23 ms and p95 latency to 2,646.53 ms with a 2.62% error rate.
For 3-Tier Swarm Cluster: The distributed topology handles 300 VUs significantly better, processing 40,735 requests with a 1.7x faster average latency (951.75 ms) and a 4.0x faster p95 latency (665.42 ms), maintaining safe performance up to 250 VUs before MongoDB Primary hits CPU limits.

Scenario 7: 12-Hour Endurance / Soak Test (`soak-test-12h.js`)

For 1-Tier Monolithic: Executing 3,429,616 requests over 12 continuous hours with 10-minute token refresh proves 0.00% heap memory leak drift, maintaining a 79.38 req/sec throughput at 130.15 ms average latency (p95: 312.95 ms).
For 3-Tier Swarm Cluster: Processing 3,561,840 requests over 12 continuous hours proves 0.00% heap memory leak drift while delivering 1.7x faster average latency (74.82 ms) and 1.9x faster p95 latency (168.35 ms) at 82.45 req/sec throughput, demonstrating superior long-term endurance stability.

Architectural Deep Dive: Why 3-Tier Outperformed 1-Tier

Hardware Resource Isolation (App Node vs DB Node):
- On 1-Tier Monolithic, all container services share a single VM's CPU scheduler and disk I/O bus. When MongoDB Primary consumed 512.2% CPU or Keycloak consumed 451.3% CPU, total VM CPU load hit 95.9% – 97.7% saturation. This caused Java Tomcat worker threads in secops-api and governance-api to stall waiting for CPU slices, driving response times to 375 ms – 575 ms.
- On 3-Tier Swarm, business APIs are isolated on the App Node (10.0.2.9), completely shielded from database disk I/O and auth encryption spikes on the DB Node (10.0.3.185). Java microservices execute instantly with sub-84ms response times (30.5 ms avg in sustained endurance).
Ingress & Gateway Offloading:
- Offloading APISIX Gateway and etcd cluster coordination to the dedicated Access Node (10.0.1.37) ensures SSL/TLS termination and URI route mapping overhead do not consume App Node CPU cycles.

Architectural Observations & Scenario Analysis

On this page