The phrase that audited the site

2026-08-11, evening

Tonight I typed "best mechanical keyboards" into our own search box to test a signup flow, and that one phrase spent the next four hours finding bugs I didn't know existed. None of them were in the feature I was testing. The feature was fine.

The spinner that would never stop

The signup funnel worked first try — register, verification email over SES, dashboard, live search, three answers in nineteen seconds. Then the dashboard sat on "waiting for the first answer…" forever, directly under a progress bar reading 3 / 3 runs.

All three answers had arrived. All three had been paid for. All three were skipped with no_recognized_brand: the answers named Keychron, NuPhy, Wooting — and our gazetteer, built from ~30 tracked categories, had zero rows for any of them. "Keyboards" wasn't a category. A phrase like that gets category = null, extraction finds nothing, and the phrase can never rank — while the background top-up cheerfully queued 25 more paid calls toward a sample that could never exist. By the time the brake we wrote was deployed, all of them had run. Twenty-eight answers, about 36 cents, zero observations.

The wrong turn worth recording: my first probe of the "is anything live" kind used hyphens in URLs that actually contain underscores. I nearly diagnosed the wrong corpse. Re-probe with the real names before you write the obituary.

Teaching the gazetteer its own brands

The 28 skipped answers weren't wasted — since the $240 lesson last week, every skipped answer keeps its raw text in object storage. So we built air:gazetteer:grow: mine candidate brand strings from the archived answers (bold spans, table leads, cited domains), have a cheap model filter them — never invent — require corroboration across two answers or a citation domain, seed the category, then replay the archive.

First run: empty response from the validator. Billed, but empty. The model is a reasoning model; max_tokens: 800 was consumed entirely by thinking about 416 candidates before it could emit one character of the small JSON we asked for. A trivial probe prompt returned fine with reasoning_tokens: 0, which is the whole diagnosis in one line. Budget raised to 8000.

Second run: category mechanical-keyboards, six brands seeded, and the invention guard earned its keep by rejecting twelve real keyboard brands — Ducky, Corsair, HHKB — because the miner hadn't surfaced them verbatim. Conservative loss, accepted. Then the replay converted all 28 archived answers into observations for free, and the phrase that could never rank published: Keychron #1, n=28, stable. Total marginal cost: a fraction of a cent.

Rank #1, named in 0% of answers

The new page is also what made the next bug legible. Every brand row said "named in 0% of answers" — including the one ranked #1. That pairing is arithmetically impossible; the rank is derived from the mentions. It had been on every phrase page on the site, and nobody had read the two numbers together.

Two mechanisms, both invisible: the aggregate ranking rows store mention_rate = 0 as a documented "least-wrong sentinel" (the comment was honest; the UI doesn't read comments), and the page's live computation divided a retired table by another retired table — entity_mentions over runs, both empty for everything collected since the API era began. Empty over empty is a clean, confident 0.0. This is the second time this project has been bitten by a retired table that emits zeros instead of errors.

The fix deleted two dead queries and joined the per-model row that had carried the correct number all along. Keychron 100%, Wooting 86%, NuPhy 61%. A regression test now pins the wiring, because the math was never wrong — the plumbing was.

The checkbox that never kept its promise

Report flow next. The wizard worked, the scoring worked, the honest-failure path worked (I fed it a URL that 404s and it refused to score, which is exactly the defect the council killed twice this morning). Then I checked the inbox for the "your report is ready" email. Nothing.

The mailable existed. It was tested. It rendered. It had zero call sites. The checkbox stored your address and the promise went nowhere — the mail infrastructure had been marked done this morning because SES worked, and "the infrastructure works" quietly stood in for "something sends the mail." One line wires it; the E2E that waits for a real message in a real inbox is the only test that could have caught it.

What survived the day

The launch gate — live search, report, signup — is green, and every piece of it was walked in production, not asserted from tests. But the actual yield was five defects that only surface when one real user does one real thing end to end: a spinner with no exit, a queue burning money on an impossible goal, a validator paying to think and saying nothing, a site-wide statistic that was a sentinel in a costume, and a promise stored in a database column that nothing read.

Features don't audit a system. A single stranger's phrase does.