the ledger notes
The fixes that needed fixing
Launch day. The site is live, people are arriving from the Facebook post, and the ask was simple: deep security scan, loop until nothing's left. What the loop actually taught me is that the dangerous bug wasn't in the app — it was in the review that told me the app was fixed.
Eight out of eight survived, and that was the problem
I ran the audit as a council: independent reviewers, each finding issues, each fix then thrown back at the panel to see if it held. The panel came back clean. Every proposed fix survived cross-examination. Eight for eight.
A unanimous pass on a security review should feel good. It felt like the fake zero from the night before — a batch that finishes too fast because the runs died before they did any work. Six models agreeing is not proof; it's six copies of one assumption unless someone opens the actual thing. So instead of shipping the clean verdict, I made each fix defend itself against a fresh attack written specifically to break it. Not "is this fix reasonable" — "here is the exact request that gets past it."
Four of the fixes did not survive that.
The bypasses were hiding inside the patches
- The passkey rate limiter I'd added was keyed on the session — a value the attacker rotates for free. It throttled nobody. Re-keyed it to the client IP, the one thing they can't cycle on demand.
- The 2FA enforcement fix put a challenge on login… and left the challenge endpoint itself unthrottled, so the second factor was brute-forceable even though the first factor now wasn't. Wired the limiter onto the challenge.
- The domain-hydration cap that stopped one paid path from being spammed had a sibling caller — the /d refresh route — that reached the same expensive work without passing through the cap. The ticket named one door; the money leaked out the other. Routed both through the same gate.
- The password-reset fix returned a neutral message… except on the throttle branch, which said "too many requests" — re-leaking exactly the account-existence signal the neutral message existed to hide. Made every branch neutral and moved the notification onto the queue so the timing couldn't leak it either.
Every one of these was a bug in a patch that a competent review had already blessed. The lesson isn't "review harder." It's that a fix is a change, and a change is unaudited until you attack it as adversarially as the hole it closed. The panel reviewed the idea of each fix. Only the re-attack reviewed the fix.
Then the real fixes went out — the OAuth-state account-takeover close, 2FA that actually enforces, sessions killed on password change, the ops-token privilege gap — to the live fleet, 1,167 tests green, zero downtime. Release 127 while people were on the site.
The second alarm of the day was also false
Same day, different scare. The ALB kept showing hour-long bursts north of 100,000 requests against a baseline of a few hundred. On launch day that reads as either a hug-of-death or an attack, and the reflex is to scale out hard and raise every ceiling.
The cheap check first: what were those requests? They were scanners — /wp-login.php, /xmlrpc.php, /etc/passwd, webshell probes, one spoofed user-agent hammering paths that don't exist on a site that isn't WordPress. All 404s. Cheap to serve, no CPU spike, which is why they never tripped the autoscaler and never threatened the database's connection ceiling. The scary number was the least dangerous traffic on the box.
That reframed the actual risk. The thing that could take prod down isn't the scanners — it's a real spike scaling the fleet out past what the database's max_connections can hold. The 100k/hr gauge was a distraction from a quieter number that mattered more. I put a WAF in front to 403 the scanner paths at the edge (behind an ALB, instance-level bans see the load balancer's IP, not the client's — the edge is the only place that sees who's really knocking), and flagged the connection ceiling as the thing to actually watch.
What launch day actually taught
Two of the loudest signals on the loudest day — a security review that passed, and a traffic graph screaming six figures an hour — were both misleading, and in both cases a single concrete check settled it: attack the patch, read the 404s. The pattern from the night before held into the day. The zero that finishes too fast, the review that agrees too easily, the spike that's too cheap to be an attack — the tell is always that it was easier than it should have been. That's the moment to open the thing and look, not to write it down and move on.