The day we stopped paying to guess

We started the day paying OpenRouter to approximate what people see when they ask ChatGPT for a recommendation. We ended it measuring the real thing, for free, and discovering that the real thing gives a different answer about a third of the time.

That last part is the business.

What we thought we were doing

The plan was four maintenance items. Add a gate so we stop publishing confidence intervals off tiny samples. Add a spending ceiling. Deepen the corpus. Go wide.

The first one turned up something worse than a missing feature. The bootstrap will happily resample twenty-three runs and hand back rank_stability_pct: 100.00 with a confidence interval of 1.000000 to 1.000000 — the most confident-looking numbers on the page, computed from a sample that cannot support them. 195 rows were doing exactly that.

Nothing was broken. Every number was computed correctly from the data it had. It was just answering a question nobody should have asked at that sample size.

That set the tone for the day: almost nothing that went wrong announced itself.

Eight dead ends, and what each one cost

Every one of these was closed by a measurement rather than an argument, which is the only reason the day was productive rather than expensive.

The prefix factorial — killed for three cents. We were about to build a corpus seven times larger to test whether "what is the best X" ranks differently from "what X should I buy". Seven ask-forms, twenty runs each, one subject: same winner every time. The keyword decides; the phrasing doesn't. Cost of finding out: $0.03. Cost of not finding out: seven times the collection budget, forever.

The Galera cluster — architecturally better, 282× slower. We found a healthy three-node MariaDB cluster already running on the Pi swarm. Synchronous replication, HAProxy failover, authenticated Redis. Strictly better than a single box on every durability axis, and our schema was fully compatible. Then we benchmarked it against our actual write shape:

single node Galera, 3-node commit p50 0.31 ms 21.38 ms batched 34,687 inserts/sec 123 inserts/sec

We needed ~395 inserts/sec. Durability would have cost two-thirds of our throughput. The architecture diagram was right and the hardware was wrong, and only a benchmark could tell us which mattered.

Three wrong diagnoses of one stall. Collection stopped at 50%. First theory: stale locks — found three, not the cause. Second: attribution jobs starving collection — the reserved jobs turned out to be all the other class. Third, and correct: 705 jobs had expired waiting in the queue. retryUntil is a wall-clock deadline from dispatch, not a budget of attempts, and it was set to an hour. A rate-limited backlog burned the whole hour without a single job running. They didn't fail. They aged out.

The fix that broke something invisible

This is the one worth dwelling on.

A brand called Drip — a real email-marketing company — was ranking #2 on every coffee query, because its alias matched the words "drip coffee". Obvious, visible, embarrassing. We fixed it by scoping brand matching to the phrase's category.

That fix then made HubSpot — mentioned in 149 of 149 CRM responses — completely invisible on CRM queries, because HubSpot's single stored category was "email-marketing."

The published CRM winners were wrong and nothing indicated a problem. A brand that fails to match looks exactly like a brand the model never mentioned.

The asymmetry is the lesson: a false positive shows up as an absurd result somebody notices. A false negative shows up as nothing at all. After you add any filter, the question isn't "did the bad thing stop" — it's "what legitimate thing did I just start dropping, and how would I ever see it?"

The real fix was scoping the ambiguous alias rather than the brand. "Drip" and "Pod" only match inside their own vertical; "HubSpot" and "Salesforce" match anywhere. CRM's #1 went from Zoho to HubSpot, where it had always belonged.

Being wrong in public, twice

We measured ChatGPT against our API and found they disagreed completely. Different #1, two brands overlapping out of seven. We wrote it up as a finding.

Then a third data point broke it, so we rewrote the explanation: divergence tracked ChatGPT's grounding source — retail listings versus editorial citations — not the category.

Then we actually measured ChatGPT's own variance and discovered it gives three different #1s in nine sessions of the same question. Which means both explanations may have been patterns fitted to noise. Both entries now carry retraction blocks rather than quiet edits.

That's the part worth keeping. A research log that only contains the conclusions that survived is a marketing document.

Where it turned

Somewhere in the afternoon the question stopped being "how do we scale our API collection" and became "why are we paying to approximate a thing we could measure directly?"

One signed-out ChatGPT session — no account, no cost — returns:

ranked brands
specific SKUs, not just brands
live prices
retailers (Best Buy, Walmart, Target)
cited publications (Tom's Guide, Consumer Reports, TechRadar)
the category's own use-case segmentation, invented on the fly
per-product justifications

Our API gives brand and position. Two fields, metered.

The publications matter most. Our schema has had a phrase_linked_domains table and a domain-score ranking since day one, both waiting for data that the API structurally cannot provide — it cites nothing. They've been empty for months. Ten browser sessions filled them:

Tom's Guide steers 9 out of 10 signed-out coffee recommendations.

Nobody publishes that. A brand can't quickly change what a model absorbed in training. It can absolutely act on knowing which publication is steering its category.

The measurement that decided everything

Ten sessions. Same question. Same machine. Same hour.

brand #1 wins Technivorm Moccamaster 6/9 — 67% OXO Brew 8-Cup 2/9 — 22% Ninja 12-Cup 1/9 — 11%

Three different winners.

This is not a flaw in the measurement. It's a property of the thing being measured, and it's the entire reason this project exists:

"ChatGPT says Technivorm is best" — refutable. Wrong about one time in three, and whoever published it cannot know whether they got the two-thirds or the one-third.

"ChatGPT names Technivorm #1 in 67% of signed-out sessions, OXO 22%, Ninja 11%, n=9" — irrefutable, and reproducible by anyone with a browser.

Every competitor screenshotting a single session is publishing noise. Not through carelessness — one observation is genuinely the most a single observation can support. The difference is knowing that, and sampling anyway.

What actually got built

The database moved off a laptop that closes, onto a machine that doesn't. Raw pages moved to object storage on the NAS — 589 KB each, and at the widest tier they'd be seven times the size of everything else combined.

Keeping those raw pages is not a nicety. When the parser turned out to be scrambling its columns, we fixed it through six separate investigations against one stored page, without touching ChatGPT again. The source is non-deterministic — re-running isn't reproducing, it's a different experiment. Without the raw capture the bug simply wasn't diagnosable.

And we proved a Raspberry Pi can do this. Playwright's bundled Chromium, headed under a virtual display, on ARM: chatgpt.com serves it normally. Eight idle Pis, no account to risk, nothing to suspend. Free parallel capacity that doesn't compete with the API's rate limit at all, because it isn't the same bottleneck.

The thing I keep relearning

Nothing that went wrong today announced itself.

Jobs that expired instead of failing. A brand suppressed rather than absent. A backfill that reported "nulled 250 rows" while leaving 100 untouched. A migration that passed its tests because the tests ran on a different database engine than production. A parser that returned eleven products, all of them wrong, sailing past a guard that only checked whether the count was zero.

Every one of them reported success.

The guards, the loud aborts, the row-count verification, the fail-closed collector — those are the product, more than any single number in the database. A ranking you can't defend is just an opinion with a decimal point.

Cost of the day: $11.24.

Coda: two more numbers, written the same evening

The emulator cost 500,000 tokens. The real hardware took 75 seconds.

The collector needed an arm64 container image for the Pi swarm. Two agents tried to cross-compile it under QEMU on an x86 host and produced nothing — a parallel-build race that silently dropped a compiled extension, a GCC internal compiler error, and a bad-relocation link failure. Three unrelated-looking bugs, each worth chasing on its own merits.

They weren't three bugs. They were one emulator. Running the identical build on a Pi that was sitting at load 0.01 finished it in 75 seconds.

The tell is worth keeping: when three different failures appear in one emulated toolchain, stop debugging the failures. Shipping the build to hardware that already matches the target feels like giving up. It's just faster.

And the fail-closed collector caught something on its first contact with production.

We deployed it to the swarm and scaled to one replica. It pulled on a node that hadn't built it, launched Chromium under a virtual display, drove a real ChatGPT session, produced an observation — and then refused to record it, because it couldn't write the raw page to object storage. The credentials weren't in the stack file. They couldn't have been: the stack file was written before the storage existed.

chatgpt-collector: exiting on batch failure (status 1)

That's the guard from this morning's post, doing exactly the thing it was built for, on a real misconfiguration, hours after being written. No partial row. No observation whose raw HTML points at nothing.

But it also exposed an ordering mistake that's worth more than the win. The expensive step — the real browser session — runs before the cheap check. So each automatic restart drove a fresh ChatGPT session and then failed at the same local write. Fail-closed and auto-restart are both correct. Composed in that order, one config gap became several wasted sessions against a resource we can't buy more of.

Cheap checks go first. The natural way to write the code is the other way around, which is precisely why it keeps happening.

AIRANKS — The Authoritative Rankings for AI Web Content