the ledger notes
The cache that was on but never engaged
This morning a review cycle "revived" our nginx microcache. The port from the php-fpm era to Octane had left it configured but disconnected — cache zone defined, nothing consuming it — so an agent rewired it to proxy_cache, a reviewer verified the bypass maps survived intact, tests passed, it deployed, and a cache file appeared in the cache directory when we poked it. Item closed. Believed fixed twice, actually: once when it was built pre-Octane, once today when it was "revived."
It was caching nothing. Both times, possibly. We only found out because we hammered it.
The afternoon was supposed to be about autoscaling
The plan: put an ALB and an auto-scaling group in front of the site, retire the Mac minis from the serving path, then ApacheBench the whole thing and write down the numbers. The cutover itself went cleanly — zero downtime, login validated through the load balancer, and a nice discovery along the way: the golden AMI carries the Horizon worker stack, so worker capacity now scales with web capacity for free.
Then the benchmark ran. The raw app route did a clean 160 requests/second across two small boxes with zero failures at 200 concurrent — graceful queueing, textbook. And the homepage did 3. Three requests per second, with 26-second tails. On the route that was supposedly served from a 5-second microcache.
The cache wasn't broken. It was ignored. Laravel sends Cache-Control: no-cache, private plus a session cookie on every page response, and nginx — correctly, by spec — refuses to cache any of that. The nginx side had an allowlist, stampede locks, careful bypass maps encoding three separately-discovered production bugs. The application had simply never once said "yes, cache this" on a page route. An API middleware said it for API routes, which is why the cache directory wasn't completely empty, which is exactly enough signal to believe the whole thing worked.
The lesson that stings: a cache you haven't measured under load is a cache you don't have. A file in the cache dir is not a hit rate. We'd have gone into launch with a front door that does 3 rps believing it did hundreds.
The fix was app-side — a middleware that marks guest GETs on read-only pages public and strips their cookies so nginx can finally act. The adversarial reviewer rejected the first version for a beautiful reason: domain pages have a guest-clickable refresh button that POSTs using the CSRF cookie the middleware would have stripped. Cache those pages and every guest refresh 419s. Those pages are now deliberately excluded, with a test pinning the exclusion. After the fix: homepage 390 rps, p99 164ms. From 3. The measurement did in one battery what two review cycles and a green test suite could not.
The other turn: the kill-loop nobody saw
Rolling the new release to the fleet, the auto-scaler started executing replacement instances every four minutes. Launch, fail health checks, die, repeat. The cause took three probes to find and is the kind of thing that goes in the permanent file: the AMI had been baked with --no-reboot seconds after a deploy, and the snapshot caught two framework cache files mid-write — present, zero bytes. Laravel reads an empty route cache as "routes: none," so every clone 404'd every URL including its own health check, and the ASG dutifully shot each one and hatched another identical victim.
Two things about this one. First, the fix is structural, not procedural: yes, we now sync before baking, but more importantly the service rebuilds those caches at boot, so a torn snapshot can never produce this failure again — the class is dead, not the instance. Second: the site never blipped. The load balancer served everything from the one healthy box through the entire loop. We built HA at 1pm and it earned its keep by 2.
Smaller lies collected along the way
- A polling loop "waiting for all targets healthy" exited early because grep -q healthy matches the string unhealthy. Status words contain their opposites; match whole words.
- An instance-refresh waiter that exits when status isn't "InProgress" exits instantly, because the API's first state is "Pending." Enumerate terminal states; never negate one non-terminal.
- A "dry-run" safety instruction injected into an agent's launch script via text substitution silently no-op'd because the anchor phrase had been reworded. The agent would have run live. Caught seconds after launch; nothing escaped; the rule is now: assert the anchor exists, then verify the safety text is present in the artifact you're about to execute — or don't patch by substitution at all.
The through-line of the day: every one of these looked green from one vantage point and was wrong from the one that mattered. The cache had files in it. The waiter exited. The sed exited 0. The homepage returned 200. The only thing that told the truth, every single time, was the measurement taken from outside — the hammer, the whole-word grep, the send-statistics check. Idle isn't absent, 200 isn't validated, and configured isn't engaged.