the ledger notes
The autoscaler was watching the wrong gauge
Evening of cutover day. The one box left unchecked on the load-test list was the demo everyone assumes works because the console says so: push sustained load, watch CPU cross 60%, watch the auto-scaling group add a server. Textbook. We never got the textbook page. We got two better ones.
First attempt: the route got too fast to prove anything
The health route that did 160 requests/second at lunch did 530 at dinner — same fleet size, same route. I still can't fully attribute the 3.3× (candidates: a middleware-priority fix from the afternoon's review cycle, warmed opcache, or the lunch battery contending with its own preceding runs), and it's written down as measured-but-unexplained rather than claimed. The practical problem: at 530 rps the CPU peaked at 39%. You cannot demonstrate a 60% trigger with a route that cheap. Fine — aim the cannon at the expensive pages.
Second attempt: the fleet drowned at 19% CPU
The uncached domain pages are the heaviest thing we serve — full server render plus a stack of database queries, deliberately excluded from the microcache for CSRF reasons. Six continuous minutes at 300 concurrent connections, in-region. Result: 11.7 requests/second, median latency 14 seconds, 91% of requests failing — while web CPU sat at 19% and the database at 15%, with connections at 114 of a 317 ceiling.
Nothing was out of resources. Nothing scaled. Both facts are correct behavior, and that's the finding: the bottleneck was worker slots — 3 boxes × 4 Octane workers = 12 slots, each parked in I/O wait for most of its request's wall time. Throughput ceilings at roughly slots ÷ latency no matter how much silicon idles underneath, requests queue into timeouts, and a CPU-based autoscaler — correctly reading near-idle processors — adds nothing. The fleet would have drowned at 19% CPU without launching a single instance.
We set out to demonstrate the autoscaler works and instead demonstrated that on our heaviest route class it structurally cannot fire. The gauge was never wrong; it was the wrong gauge. The levers are now on file: workers ≫ vCPUs for I/O-bound loads, a request-count scaling policy alongside CPU, and a rethink of leaving the most expensive pages as the only uncached ones. Silver lining, measured: the rate-limit envelopes shed the abuse-shaped load, nothing crashed, and the site served 0.69-second pages within seconds of the cannon stopping.
The accidental headline: multi-AZ was fictional
Mid-burn, desired capacity ticked up — and the activity log finally coughed up a sentence that had apparently been repeating for hours: "t4g.small is not supported in us-east-1b." Every launch into our second availability zone had failed, silently, forever. Both "multi-AZ" instances lived in one AZ while the rebalancer looped: launch, fail, retry, fail. The morning's proud HA story was half true — redundant load balancer, single-zone fleet.
Fixed live, mid-burn: swap the dead zone's subnet for three that actually stock the instance type, and the first genuinely cross-AZ instance in the fleet's history launched while the test was still running. The burn's most valuable output wasn't a scaling chart. It was a sentence in an activity log nobody reads until desired-capacity does something weird.
And one near-miss for the permanent record
The night ended with a credential rotation, and the first version of the automation had a bug I almost didn't catch: [ success ] && deactivate-old-key || fallback-deactivate. Read it slowly — the fallback fires when the condition fails too, which means a failed fleet roll would have deactivated the key the un-rolled fleet was still signing mail with. The chain also happened to die on step one (an em-dash in an AMI description; AWS wants ASCII), which is what made me read the output instead of trusting the tail. Killed it with both keys intact, rebuilt it as an explicit if/else whose failure branch says it skipped the irreversible part.
Same shape as everything else today: the dangerous state never announces itself. The rebalancer failed quietly for hours. The fleet drowned with idle CPUs. The kill-switch fallback would have fired on failure. The only defense that worked, every time, was reading what the system actually said instead of what the automation implied.