MCM Load Testing & Performance Benchmarks

This section provides comprehensive performance benchmarks, architectural capacity limits, and infrastructure sizing recommendations for the Multi-Cloud Manager (MCM) platform, comparing single-node 1-Tier Monolithic vs distributed 3-Tier Docker Swarm deployment topologies.

Understanding Performance Load Testing & Key Metrics

Performance load testing measures system stability, transaction throughput, and response latency profiles under simulated concurrent user traffic. Benchmarking helps identify hardware bottlenecks, database connection ceilings, and microservice threading limits.

What is Load Testing?

Load Testing is a core performance engineering methodology designed to measure system stability, transaction throughput, and response latency distribution under simulated operational traffic. By generating concurrent Virtual User (VU) requests against a system, load testing evaluates how API gateways, application microservices, database connection pools, and operating system schedulers handle peak operational stress.

End-to-End Load Testing & Execution Flow:

Load Test Execution Pipeline:
  [Virtual Users (VUs)] ──► [HTTP API Endpoints] ──► [Microservices & DB] ──► [Resource Tracking & Threshold SLA Checks]

Why Load Testing is Done:

• Identify System & Architectural Bottlenecks: Uncovers CPU saturation, memory leaks, database lock contention, connection pool exhaustion, and thread starvation before deploying to production.
• Hardware & Capacity Sizing: Provides empirical performance benchmarks to determine optimal hardware specifications, container resource limits, and node topology scaling (e.g. 1-Tier Monolithic vs 3-Tier Distributed Clusters).
• SLA & SLO Compliance Validation: Validates that system throughput (RPS), average latency, 95th percentile (p95) response times, and HTTP error rates strictly comply with target Service Level Agreements (SLAs).
• Long-Term System Stability: Audits long-duration continuous traffic resilience (soak testing) to detect gradual JVM heap drift, connection pool decay, and memory leaks.

Tools Used for Load Testing & Benchmarking:

• Grafana k6 (Primary Load Generation Engine):
  Grafana k6 is a modern, high-performance, developer-centric load testing tool written in Go with JavaScript ES6 scriptable execution.

  • Why k6 Was Chosen:
    • High-Performance Go Engine: Written in Go, k6 handles thousands of concurrent Virtual Users (VUs) per machine with an extremely low CPU and RAM footprint compared to legacy Java-based tools like JMeter.
    • Developer-Friendly JS Scripting: Allows writing load test scenarios, custom VU iteration loops, setup hooks, and authentication flows directly in modular JavaScript.
    • Flexible Concurrency & Traffic Profiles: Supports dynamic VU ramping, spike patterns, constant arrival rates, and long-duration soak schedules.
    • Built-in Thresholds & SLA Assertions: Automatically validates pass/fail SLA criteria (e.g., p95 < 1500ms, error_rate < 1.00%) directly inside the execution pipeline.

• Docker Stats CLI (Container Resource Monitoring):
  Native Docker command line monitoring utility used to collect real-time container metrics (CPU utilization percentage, RAM memory usage, network throughput, and block I/O) across active system containers during test execution.

Core Key Performance Indicators (KPIs) Explained:

• Throughput (Requests Per Second - RPS):
  The number of successful HTTP API requests processed by the platform per second. Measures overall system capacity. (SLA Target: > 10.0 req/sec).

• Average Response Time (Latency):
  The average time (in milliseconds) taken for microservices to process and return data for an API call. (SLA Target: < 500 ms).

• 95th Percentile Latency (p95):
  The latency boundary within which 95% of all requests complete. This is the primary metric for evaluating user experience under worst-case peak traffic. (SLA Target: < 1,500 ms).

• HTTP Error Rate (% Failures):
  The percentage of total requests that fail due to server errors (5xx), connection timeouts, or authentication lockouts (403). (SLA Target: < 1.00%).

• Container CPU & Memory Utilization:
  Real-time hardware resource consumption tracked across all Java microservices, Keycloak, MongoDB, and Elasticsearch containers using docker stats.

Iteration Workflow: What We Do in Each Loop & Why

In typical enterprise operations, human IT operators log into Multi-Cloud Manager and perform end-to-end multi-module workflows (auditing cloud inventory, checking cost allocations, reviewing security threat posture, and auditing governance compliance policies).

To mirror this authentic operator journey, every Virtual User (VU) iteration executes the following sequential steps:

1. Step 1: Discovery Search (/discovery-api/cloud-resources/search):
   Queries indexed AWS, Azure, and Docker Hub resource inventories.
2. Step 2: FinOps Cost Analytics (/finops-api/cloud-cost/labels):
   Retrieves cost distribution breakdown across cloud tags and labels.
3. Step 3: SecOps Vulnerability Search (/secops-api/vulnerabilities/search):
   Queries active threat scans, CVE records, and Wazuh/Falco security alerts.
4. Step 4: Governance Compliance Search (/governance-api/policies/search):
   Evaluates active compliance policies and rule group execution statuses.

Scope Rationale: Module Selection

The load testing suite explicitly targets the 4 core operational search modules (Discovery, FinOps, SecOps, Governance). These modules represent over 95% of daily interactive user query volume across Multi-Cloud Manager dashboards.

Token Reuse Methodology: Where & Why Applied

To accurately simulate real-world user behavior and isolate microservice capacity, the test suite implements two distinct authentication patterns:

Pattern A: Setup-Phase Token Reuse (Used in 5/7 Tests)
  k6 setup() ──► POST /api/auth/login ──► Stores JWT Tokens in authMap{} ──► VUs execute microservice APIs with "Authorization: Bearer <Token>"

Pattern B: Per-Iteration Auth (Used in 2/7 Tests)
  VU loop ──► POST /api/auth/login ──► Keycloak PBKDF2 BCrypt Hash ──► Executes microservice APIs ──► Logout

1. Where Token Reuse Was Applied:

• peak-stress-test.js (100 VUs, 3m): Pre-authenticates in setup() and reuses JWT Bearer tokens across all 100 VUs.
• spike-test.js (50 ➔ 120 ➔ 50 VUs, 3.5m): Pre-authenticates in setup() and reuses tokens during the 120 VU traffic surge.
• max-api-capacity-test.js (50 ➔ 300 VUs, 4m): Pre-authenticates in setup() to isolate pure API query limits.
• sustained-load-test.js (50 VUs, 15m): Pre-authenticates in setup() and reuses tokens for 15 minutes continuous execution.
• soak-test-12h.js (50 VUs, 12 Hours): Implements 10-Minute Token Auto-Refresh: each VU reuses its token for 10 minutes (600,000 ms) before fetching a fresh token, preventing JWT expiration without logging out.

2. Why Token Reuse Was Necessary:

• Real-World Fidelity: Human users in a modern Single Page Application (SPA) web frontend log in once at the start of a session and reuse their JWT access token for all subsequent navigation and dashboard queries.
• Isolating Microservice Capacity: Keycloak's PBKDF2/BCrypt password encryption is intentionally CPU-heavy. Logging in on every single API iteration causes Keycloak to consume 350% – 450% CPU, starving database connection pools and causing false 403 Forbidden lockouts. Token reuse dropped Keycloak CPU load to < 15%, allowing microservices to achieve 274 req/sec at 30.5 ms average latency.

Comprehensive Load Test Suite Breakdown

The platform was subjected to 7 standardized load testing scenarios:

1. Baseline Multi-Module Performance (enterprise-modules-test.js):
   • Profile: 50-100 VUs.
   • Flow: Evaluates clean-state API search processing across all major platform modules (discovery, secops, finops, governance) under standard concurrent user load (50 VUs for 3 minutes).
2. Peak Daily Stress Test (peak-stress-test.js):
   • Profile: 50-100 VUs.
   • Flow: Pushes microservices to peak daily concurrency (100 VUs for 3 minutes) using setup-phase token pre-authentication to isolate Java microservice CPU capacity from auth overhead.
3. Traffic Spike Tolerance (spike-test.js):
   • Profile: 50-100 VUs.
   • Flow: Simulates a sudden traffic surge from 50 to 120 VUs within 15 seconds over 3.5 minutes using token reuse to verify if request queues drain gracefully without memory leaks or service crashes.
4. Sustained Endurance Test (sustained-load-test.js):
   • Profile: 50-100 VUs.
   • Flow: Sustains continuous load at 50 VUs for 15 minutes with token reuse to verify long-term memory stability and absence of memory leaks.
5. Max Auth Capacity Test (max-auth-capacity-test.js):
   • Profile: 50-100 VUs.
   • Flow: Ramps continuous login operations up to 100 VUs over 3 minutes (without token reuse) to determine Keycloak authentication CPU limits.
6. Max API Capacity Test (max-api-capacity-test.js):
   • Profile: 50-100 VUs.
   • Flow: Ramps pure API query transactions from 1 to 300 VUs over 4 minutes using token reuse to find the absolute safe database and microservice transaction ceiling.
7. 12-Hour Endurance / Soak Test (soak-test-12h.js):
   • Profile: 50-100 VUs.
   • Flow: Runs continuously for 12 hours at 50 VUs with 10-minute token auto-refresh to audit long-term heap memory stability, connection pool degradation, and container uptime across 3.5M+ API calls.

Detailed Documentation Sections

Explore the full benchmark reports, hardware sizing calculators, and comparative architectural analysis:

• Sizing & Performance Report
  Detailed scenario-by-scenario metrics breakdowns, container CPU/RAM utilization matrices, and performance limits.

• Architecture Comparison (1-Tier vs 3-Tier)
  Side-by-side performance analysis comparing single-node monolithic VM vs distributed 3-Tier Docker Swarm topologies.