Autoscale + Load Test — When Your Website Needs to Handle 10,000 Concurrent Users, How Do You Know It Actually Can?

The Problem: "The Website Crashes Every Time We Launch a Campaign"

A client came to us with a problem that's painfully familiar to many engineering teams — their web application would crash every time marketing ran a campaign. On normal days, the system handled traffic comfortably. But when traffic surged 10–50x within five minutes, the servers buckled.

The pattern repeated like clockwork:

• Response times jumped from 200ms to 15 seconds
• Error rates blew past 40% within the first three minutes
• Marketing had already spent the ad budget — but customers couldn't even load the page
• The DevOps team would scramble to manually scale servers, a process that took 20–30 minutes

The cost: Each 30-minute outage resulted in an estimated ฿500,000–2,000,000 (~$14,000–$57,000) in lost revenue, depending on campaign size.

Defining Success

We aligned with the client on crystal-clear targets:

The hardest part wasn't "making it scale." It was proving it could scale — before the real campaign went live.

Architecture Principles

Scale Horizontally, Fail Gracefully

We designed the architecture around three core principles:

1. Stateless Application — Every instance must function independently with no internal state, enabling instant horizontal scaling
2. Metric-driven Scaling — Decisions based on real signals, not just CPU utilization — including request queue depth and response time
3. Graceful Degradation — If the system truly can't keep up, it degrades in a controlled manner rather than collapsing entirely

Multi-layer Scaling Strategy

We didn't just scale the application servers. Every layer was designed to scale independently:

• Load Balancer Layer — Traffic distribution and health checking
• Application Layer — Scales based on request rate and response time
• Cache Layer — Absorbs read-heavy traffic so not every request hits the database
• Database Layer — Read replicas for queries that don't require 100% real-time data
• Queue Layer — Offloads non-urgent work (emails, analytics updates) to asynchronous processing

Scaling Triggers: Beyond CPU

Many organizations set autoscaling to trigger at CPU > 70% — sounds reasonable, but in practice it's far too slow. By the time CPU hits 70%, users are already experiencing timeouts.

We used Composite Metrics that combine multiple signals:

• Request Rate — Requests per second (a leading indicator)
• Response Time P95 — If it starts climbing, the system is about to struggle
• Active Connections — Number of connections waiting for a response
• Queue Depth — Number of tasks waiting in queue

When these signals cross their thresholds simultaneously, the system scales up immediately — without waiting for CPU to overheat.

Predictive Scaling: Knowing Before the Spike Arrives

For campaigns with known schedules, we added Scheduled Scaling — the system pre-warms instances 15 minutes before a campaign goes live. No waiting for traffic to arrive before reacting.

Performance Testing: Proving It Before Launch Day

This is the part we care about most — if you don't test properly, you're gambling.

Test Strategy: Four Layers

We designed testing in four progressive layers, each with a distinct purpose:

1. Baseline Test — Know Your Starting Point

Test at normal traffic levels (500 concurrent users) to establish baseline performance. If the baseline is poor, scaling won't help — you'll just have more slow instances.

2. Load Test — Test the Target

Gradually ramp from 500 to 10,000 concurrent users over 20 minutes to observe:

• How quickly does the autoscaler respond?
• How fast are new instances ready to serve traffic?
• Do errors occur during scaling transitions?

3. Spike Test — Simulate Reality

This is the most critical test. Simulate traffic jumping from 500 to 8,000 within 60 seconds — mimicking a real campaign launch. The key questions:

• Can the system survive the first 60–120 seconds before new instances are ready?
• Are any requests timing out while waiting for scale-up?

4. Soak Test — Test Endurance

Run 10,000 users continuously for 4 hours to detect:

• Memory leaks
• Connection pool exhaustion
• Database connections not being returned
• Cache invalidation behaving unexpectedly

Designing Realistic Test Scenarios

A common mistake in load testing is hammering a single endpoint repeatedly — the result is a 99% cache hit rate and beautiful numbers that mean nothing in production.

We designed test scenarios that mirror actual user behavior:

Each scenario includes Think Time (the pause while a user reads a page before clicking) randomized between 2–8 seconds, closely mimicking real behavior.