the ledger notes
The audit that audited the auditor
Four hours before an investor demo, we ran a council review of everything public: every page, every form, every number a technically-literate stranger might poke. The council found what councils find — an unbranded 404, a www host that never redirected, honest zeros that needed a sentence of explanation. All fixed within the hour.
The interesting part is what the review process itself shook loose, because three of the day's four serious defects were introduced by our own fixes.
Alienware, ranked #1 and #13 simultaneously
A ranking page showed the same brand at two positions — Alienware #1 and #13, Dell #2 and #9, five doubles in one visible list. The mechanism: our brands table is per-category, so "Dell" exists once for gaming monitors and once for laptops, and a monitor phrase's mentions were attributed to both rows. One measurement, counted twice. The tell that it was double-attribution and not two real signals: the off-category twin always carried NULL stability, because the bootstrap had nothing real to resample.
The fix collapses category twins at the one service every ranking consumer routes through — best-ranked row keeps its metrics, ranks resequence, and only then does the top-N cut apply, so affected lists backfill instead of showing holes.
The cache purge that took the site down
Yesterday's microcache made deploys dangerous in a way nobody noticed: cached HTML referenced the old release's hashed assets, which the deploy had just deleted. So we added a purge step.
The purge deleted cache files under a running nginx. Its shared-memory key zone still believed those entries existed; fastcgi_cache_lock then wedged every cacheable path into 60-second 504s — while uncached paths served fine in 0.6s and load average sat at 0.11. A site that is simultaneously up and down, depending on which URL you ask.
Restart-after-wipe fixed it. The zone only rebuilds on restart; there is no safe way to delete under it.
The homepage that could DoS itself
Restoring service meant clearing the app cache — which evicted the homepage's count-up band. One of those counts is COUNT(DISTINCT url) over the observation-sources table: minutes of query at current scale. Nothing stopped requests from computing it inline, so every homepage hit fired its own copy. Eight stacked five-minute queries ate every php-fpm worker.
The numbers band now never computes on the request path. A scheduled command warms it hourly; a cache miss renders an absent band. An absent band is a better failure than a dead site, and "better failure" is a choice you have to make on purpose.
The part that worked
The morning's probe — 25 questions about ourselves through the Machine's own search — had a delayed payoff: when we regenerated our domain summary during the review, the Machine's browsing read our current pages and wrote back an accurate, even flattering description. Two days ago it said our homepage was "essentially no substantive content."
Every domain we know about — 8,724 — has now been through the same pipeline the toolbar triggers, most-cited first, because the daily summary budget means dispatch order decides who gets described today.
Total for the afternoon: eight deploys, one self-inflicted twelve-minute outage, 890 tests green, and a public surface we'd hand to anyone. The council's real function wasn't the list of findings — it was forcing the kind of adversarial walkthrough where your own last three fixes get treated as suspects. They were guilty.