The data was there the whole time

Earlier today I wrote about the moment we stopped paying to approximate ChatGPT and started measuring it directly. This is the second post of the same day, which breaks my own rule about not writing one unless something turned.

Something turned.

We spent the evening designing an elaborate accounting system for phrases that "can never produce data." Then we opened one of the pages and found the data.

The failure that looked like an answer

The collector fails closed with no_products_extracted. On the swarm it hit that on phrase id=2, "best crm for small business", and — because a fail-closed trigger stopped the entire run — killed a batch of 1,461 queued sessions on all three nodes it landed on.

The obvious reading: ChatGPT answers software questions with prose, we need product cards, so roughly 370 of 1,463 phrases (crm, vpns, web-hosting, credit-cards, project-management) are structurally incapable of yielding anything.

That reading survived for hours. It shaped a fix, a design brief, and a whole workflow about how to account for permanently-unparseable inputs without re-collecting them forever.

It was wrong.

What was actually in the page

One live session. One saved file. 599KB of HTML.

Plus an ordered "my quick picks" list naming the same five brands against different segments. Plus six citation links — hubspot.com, pipedrive.com, salesforce.com, and techradar.com.

A crude grep for brand names in a page reported as containing zero products: HubSpot ×13, Salesforce ×19, Pipedrive ×7, Zoho ×3, Freshsales ×2.

The extractor wasn't measuring the source. It was measuring its own coverage of the source. Those are different claims, and only one of them requires work.

Four shapes, not two

Three more sessions, three more saved pages:

Shape Categories Structure Carousel coffee, footwear product cards — brand, price, retailer Table crm CRM | Best for | Why it stands out Table credit cards Card | Gas cash back | Best for Medallist vpns no table at all — 🥇🥈🥉 headings

Two details that decide the whole design:

The tables aren't positionally consistent. "Best for" is column 1 for CRM and column 2 for credit cards, with a domain-specific column beside it. Any extractor that counts columns breaks on the second category it meets. The column has to be found semantically.

The medallist shape asserts a ranking the others don't.

🥇 Best overall budget pick: Surfshark 🥈 Best privacy-focused budget option: Proton VPN 🥉 Best if you want premium features for less: NordVPN

Reading a rank off a medal is measurement. Reading one off table row order would be inference dressed as measurement — and inventing a rank the answer never asserted is the single thing this project cannot survive doing.

And every shape carried citations: 6, 5, and 11. Including TechRadar, NerdWallet, Bankrate. Our phrase_linked_domains table has been empty for months waiting for exactly this, and the paid API can never supply it, because the API cites nothing.

The categories we'd written off as empty are the ones carrying the richest data in the system.

The bug I shipped, and the measurement that caught it

The first fix was to stop halting the whole batch on an answer-level failure — skip that phrase, keep going. That needs a backstop, or a broken parser silently skips all 1,461 and looks exactly like a clean run that collected nothing.

The backstop I wrote: halt after 5 consecutive skips. The reasoning felt airtight — content variation is scattered, a broken parser is consecutive.

Then I measured the corpus instead of trusting the reasoning:

longest consecutive run of software-category phrases : 174 runs of >= 5 consecutive : 10

Phrases are ordered by primary key and grouped by category. A threshold of 5 trips ten times in a single pass. And the container's entrypoint is while true — so each halt restarts the batch, which reaches the same cluster, and halts again.

My fix for "aborts on the first bad phrase" would have quietly degraded into "aborts forever."

"Consecutive failures mean a systematic problem" is a near-universal heuristic, and it silently assumes the input is randomly ordered with respect to the failure cause. Real inputs almost never are. They come from ORDER BY id, directory listings, alphabetical sorts, category groupings — any of which can correlate perfectly with why an item fails.

The replacement keys on a different question: has this batch banked any success at all? One success proves the machinery works, so every later failure is input variation by definition, and no amount of clustering can trip it. A genuinely broken parser produces zero successes however long it runs. The regression test walks 300 consecutive skips after one success and asserts none halt.

Why any of this was recoverable

Because the pages were saved.

The whole diagnosis — four shapes, the column inconsistency, the medal ranking, the citation counts — came from files on disk. Zero additional sessions. And that matters more here than in a normal system, because re-running is not reproducing. The same question to the same model in the same hour returns a different answer; we measured 3 different #1 brands across 9 identical sessions. A re-fetch is a new experiment, not a second look at the old one.

So the raw capture isn't an archive. It's the only artifact that can be examined twice.

Which made the actual bug embarrassing in a useful way: we were failing closed and destroying the page as we did it. The one artifact that distinguishes "this phrase has no products" from "our parser is blind to a layout" was being thrown away at exactly the moment it became interesting. Skipped captures now go to object storage — 109KB gzipped from 608KB raw — keyed by phrase and reason, and deliberately not as observation rows, because a row with no products would publish a phrase as though the model had recommended nothing.

The reframe worth more than the fix

The segment text — "Best overall for many small businesses", "Budget-conscious sales teams", "Best privacy-focused budget option" — currently lands in a varchar(255) column called use_case_segment. Free text on a row. There is no attributes table anywhere in the schema.

Making those first-class entities changes what the product can say. We already compute bootstrap confidence intervals and rank stability over repeated sessions. Point that machinery at (brand, attribute) pairs instead of just brands, and the output becomes:

"HubSpot owns 'best overall for small business' in 67% of signed-out sessions, n=9, CI 41–86%."

A brand cannot quickly change what a model absorbed during training. It can absolutely act on knowing which segment it owns, and which competitor owns the one it wants. That's a more useful question than "am I #1", and nobody in the category publishes it.

We had been collecting the raw material for that since the first session. We were writing it to a string column and calling the pages that contained the most of it "empty."

What actually shipped

A Raspberry Pi drove a real ChatGPT session and produced a complete observation:

#1 Asics — ASICS NOVABLAST 6 ($154.99 @ DICK'S Sporting Goods) #2 Hoka — HOKA Clifton 10 ($123.99) #3 Nike — Nike Pegasus 42 ($144.99) raw HTML in object storage: 598,099 bytes

Five products with prices and retailers, the page archived, the row in the database on a machine that isn't a laptop. The collector works. It was never the bottleneck.

Also landed: five reviewed fixes, including one that had been silently truncating every multi-word brand in the gazetteer — Land Rover Defender → "Land", Bank of America Customized Cash Rewards → "Bank". 112 brands unreachable, in the exact column the rankings group on. Verified afterward on data it was never developed against: American Express Blue Cash Everyday now resolves to American Express.

The collector stays at zero replicas tonight. The accounting that stops it re-collecting those 370 phrases forever doesn't exist yet, and while true is not a thing to leave running against someone else's website on a hunch.

The lesson, restated

This morning's post ended on "nothing that went wrong announced itself." Tonight's is narrower and sharper:

A zero from your extractor is a measurement of your extractor.

The tell is concentration. Genuine emptiness scatters. Coverage gaps cluster — by category, by file type, by language, by whatever varies the shape you didn't implement. When a failure reason phrased as absence lines up neatly with a subset of your inputs, open one of them by hand before you build anything on top of the conclusion.

We built a design for accounting for data that didn't exist. It took one file, and about ninety seconds of reading it, to find out the data existed.

AIRANKS — The Authoritative Rankings for AI Web Content