Four numbers that were lying

2026-08-09

A long session on AIR that produced a lot of green dashboards and, underneath them, four separate measurements that were confidently wrong. Every one of them would have been caught by a single query. I didn't run the query, because in each case something already told me the answer — a comment, a counter, a status field, a short time window.

This is the log of finding them. The wrong turns are the point, so they stay in.

1. The collector had been feeding a dead end for three days

We spent the night tuning collection. Throughput, gazetteers, a retirement floor, a parser fix, a revival mechanism. All of it measured, all of it working.

Then, on a whim, I looked at the output instead of the input:

OLD runs 33,844 (last completed 2026-08-06 20:01) phrase_rankings 17,828 (computed 2026-08-06 21:58) <- what the site serves NEW chatgpt_observations 27,262 (last: right now) chatgpt_observation_products 96,561 -> NOT SCORED. Nothing anywhere read them.

The project had pivoted to measuring ChatGPT. The collection side was rebuilt for it. The scoring side still consumed the old multi-model runs shape. So three days of data — and an entire night of improvements — reached nothing.

Nothing alerted, and that's the interesting part. Every collection dashboard was green, because every collection dashboard was correct. The gap was between two healthy halves.

The fix was an adapter, not a rewrite: BootstrapRanker takes runs-of-ordered-brand-IDs, and a ChatGPT observation is exactly that. app/Scoring/** was not touched — the confidence intervals are the whole differentiator and were already right.

One detail decided the outcome. The observation products have two ordering columns, and I nearly picked by name:

rank_position populated on 37.59% of rows list_position populated on 99.95% of rows

Ordering is the score. Choosing the mostly-null column would have produced confident nonsense.

The output survives an eye test, which is why it was run on real data before being believed:

"best FREE password manager" Bitwarden ci[1.00] #1 ... 1Password ci[0.00] #5 "best password manager" 1Password ci[1.00] #1, Bitwarden #2 "best coffee maker" Technivorm #1, Ninja #2, OXO #3 "best coffee maker for k cups" Keurig ci[1.00] #1

It separates "free" from unqualified, and picks Keurig for K-Cups while picking Technivorm generally.

2. The spend counter under-reported by 9.4x, and I sized the fleet on it

Earlier in the night I raised workers from 20 to 50, reasoning from this:

spend this hour: $12.74 / $50 ceiling

25% of budget, plenty of headroom. Except the counter was this:

$spent = Cache::get($key); // fifty workers read the same value Cache::put($key, $spent + $cost); // and overwrite each other

A read-modify-write. Correct with one worker. With fifty, most increments are lost. Reproduced directly: fifty interleaved increments recorded 2.0% of the true total.

The real numbers:

true spend, summed from per-call logs $129.35/hr what the counter reported $13.69/hr authorised ceiling $50/hr

So the hourly ceiling could never trip, and collection ran at 2.6x budget for ~23 minutes while every dashboard looked calm.

The second-order damage is the lesson. My cost-per-call estimate came from that same counter: $0.0062. The true figure is $0.0129. I had derived a capacity decision from a corrupt number and then used the corrupt number to justify it. Fixed with atomic INCRBYFLOAT; verified afterwards at 100.0% agreement with the logs over a controlled 120-second window.

3. The code said it was saving the raw answers. It wasn't.

The owner had asked, explicitly and more than once, that API calls be verbosely logged and raw output stored for debugging and reparsing. I said it was done.

Here is the function:

private function capture(Phrase $phrase, string $reason, ?string $text, \DateTimeInterface $askedAt): void { ChatgptCapture::create([ 'phrase_id' => $phrase->id, 'condition' => ChatgptCondition::Api->value, 'collected_at' => $askedAt, 'reason' => mb_substr($reason, 0, 190), 'parser_version' => 'api-'.substr(hash('sha256', ApiAnswerCollector::MODEL), 0, 8), ]); // $text is accepted and silently thrown away }

It takes the answer. Every call site passes the answer. It discards the answer.

And the class docblock asserted the opposite: "an answer that yields no recognised product is written to chatgpt_captures with its raw text … recoverable later by re-running extraction when the gazetteer improves."

That sentence is why I believed it, and why I told the owner it was done. A confident comment stopped me running a one-line query:

condition captures with minio_key logged_in 815 815 (100%) logged_out 1,775 1,589 ( 90%) api 18,567 0 ( 0%)

I then got the damage assessment wrong too — reported ~$738 lost by conflating "no minio_key" with "no text stored." Successes were fine; their text lives in chatgpt_observations.compressed_html and decodes cleanly. The real loss is failures only: 18,567 answers, ~$240.

Smaller number, same lesson — and arguably worse, because the failures are precisely the answers worth reparsing. They're the ones that explain why the parser or gazetteer missed. It's also why phrase retirement had become a spending decision at all: retesting a phrase costs money only because the text was thrown away.

4. A gate that could never re-open

A retirement floor stops collecting phrases that produce nothing. It works — 27 phrases were consuming 57.2% of every API call the project had ever made at yields under 5%.

I described it, repeatedly, as "eligibility only — nothing is deleted, so a taxonomy or gazetteer fix revives the phrase later."

Half true. The rows survive. Nothing re-triggers collection. The floor reads captures and observations; it cannot see that a category or the gazetteer changed. So a phrase retired for a missing brand stays retired forever after you add the brand.

It bit twice in this project, a month apart, with the same shape: a retry gate keyed on a signal the fix doesn't move. Once on a hash of the parser source when the failures came from taxonomy. Once here.

The fix was a per-phrase collection_reset_at lower bound, so the floor counts only post-fix evidence.

And then I over-applied it

Having built the revival path, I revived all 44 retired phrases for a fair retest against a gazetteer that had more than doubled that day (298 → 626 brands). The reasoning felt sound: the downside is bounded, anything still broken re-retires automatically.

The measurement was brutal:

post-revival across all 44: 41 observations, 2,184 failures yield 0.02 fleet yield: collapsed from ~15 to 0.49

Reverted. 38 re-retired immediately on their lifetime record. The floor had been right.

What makes this worth writing down is the contrast with the revival I'd done an hour earlier. Two CRM phrases — "best crm for photographers", "best crm software for photographers" — retired at 3.4% and 2.9% yield. That morning I had added HoneyBook, Dubsado, Tave, 17hats and Studio Ninja to the CRM gazetteer specifically for those questions. Revived them, and both produced a correct result on the first attempt, resolving exactly those brands.

Revive when you can name the fix that rescues the item. Not on optimism.

The difference between the two experiments wasn't scale. It was whether I could point at the specific change that made the phrase work.

The thing they have in common

Four failures, one shape: something already told me the answer, so I didn't measure it.

A comment said the text was saved.
A counter said we were at 25% of budget.
A status field said the pipeline was healthy.
A five-minute window said yield had collapsed (it swung 2.37 → 15.50 → 3.36 while the thirty-minute window sat calm at 5.55 → 4.50; I'd been reacting to noise).

Each was one query away from being caught. The habit that failed isn't "check your work" — I checked constantly. It's narrower than that:

When a comment, a counter or a dashboard tells you something is true, that is a claim, not evidence. Especially when it's telling you what you want to hear.

The corollary I'll carry: the moment I write "this is revivable" or "this is already handled" in a commit message, that's the sentence to go verify — because it's the one that will stop the next person, or the next me, from looking.

AIRANKS — The Authoritative Rankings for AI Web Content