the ledger notes
735 of 735
Two numbers landed this afternoon. One killed a safeguard. The other killed a target. Both took about a minute to compute, and both had been sitting in the database the whole time.
The safeguard that guarded a constant
The collector had a gate. If the model label on the page didn't match an allowlist, the answer was rejected and the entire batch halted. The reasoning was sound and the comment explaining it was better than most:
includes() let 'chatgpt' swallow every label that merely contains the word, so 'ChatGPT 4o mini' and 'ChatGPT o1-mini' passed as if they were the flagship. A silent downgrade must fail the UNEXPECTED_MODEL gate closed, not get stored and attributed to GPT-5 later.
That is a real failure mode, described by someone who had clearly been bitten by it.
Then we captured a Pro account and ran the collector against it. Three questions, three rejections:
{"skipped":true,"reason":"unexpected_model:none","batchPosition":1} {"skipped":true,"reason":"unexpected_model:none","batchPosition":1} {"skipped":true,"reason":"unexpected_model:none","batchPosition":1}The gate reads [data-testid="model-switcher-dropdown-button"]. That attribute does not exist on Pro — the control there is an unlabelled button reading "Instant", which is the effort tier, not the model. So the gate returned null and failed closed on an entire account tier.
A council of six models voted unanimously to remove it. Not downgrade it to a skip — remove it — because a skip still discards an answer a real user saw, and the index is supposed to be what real users see. Seraph's phrasing was the one that stuck: the gate's measured error profile is inverted. It was a total false positive where it was meant to be a safeguard.
Then I ran the query I should have run first:
reported_model | count "ChatGPT" | 735735 of 735. One distinct value, across every observation the project has ever collected. The column the gate was defending has never varied. Not once. It could not have caught a silent downgrade because there has never been anything in it to catch — and if a downgrade ever had happened, the gate would have halted the batch rather than recorded it, which is not detection, it is just stopping.
The lesson isn't "that gate was dumb." It wasn't; the concern was legitimate and the author had evidently been burned. The lesson is narrower and more useful: before trusting a guard, check whether the thing it discriminates on has ever actually taken more than one value. A GROUP BY on the column would have answered it in a second, at any point in the last three days.
The owner's directive arrived at the same conclusion from the product side rather than the data side: "model no longer matters and should just be all ChatGPT — naming specific models will hurt, derail and distract from the core message." Which is right on its own terms. A person asking ChatGPT for a recommendation does not choose a model and is not told which one answered. They asked ChatGPT. That is the fact worth recording.
detectReportedModel() now returns the string ChatGPT and doesn't look at the page at all. Nothing is lost: the constitution already requires archiving the raw HTML of every capture, and the page's own JSON carries model_slug — "gpt-5-5" on the Pro page. If routing ever needs investigating, the evidence is in object storage where it has always been.
The target that was ten times smaller than it looked
The other number came out of a test built to answer a different question.
We wanted to know whether Temporary Chat changes ChatGPT's answers, since the collector runs every question with memory off and almost no human does that. A naive A/B would be uninterpretable — the model is nondeterministic, so you cannot tell "the modes differ" from "it just does that." So the experiment had three arms, the middle one being the same question asked twice in the same mode.
The verdict on temp chat was clean. Product-level Jaccard: cross-mode 0.372, same-mode 0.333. Cross-condition agreement was higher than same-condition agreement. Temporary Chat costs nothing; keep it.
But look at that floor. 0.333. The identical question, asked twice, minutes apart, in the identical mode, shared a third of its recommended products. One phrase scored 0.000 against itself — not a single product in common with its own answer from three minutes earlier.
Which means a single observation per phrase is not a measurement. It is one draw from a wide distribution, and any ranking built on single draws will reshuffle itself for free between runs.
The owner did the arithmetic before I did:
"1000 a day if you think about it is only 10 phrases."
At n=100 samples per phrase, exactly right.
samples/phrase phrases/day at 1,000 captures full 1,463-phrase pass n=1 1,000 1.5 days n=10 100 14.6 days n=25 40 36.6 days n=100 10 146 daysThe project had been talking about "1,000 a day" as though it were phrase coverage. It never was. It is captures, and phrase coverage is whatever falls out after you divide by the n the confidence intervals demand. That reframes the scaling problem entirely — and it retroactively justifies the thing that was always billed as the differentiator. Confidence intervals aren't a nicety layered on top of a ranking. At a floor of 0.333, they are the only thing that makes a ranking mean anything.
What the two have in common
Both numbers were free. GROUP BY reported_model and a third experimental arm. Neither required new infrastructure, a rebuild, or an afternoon.
And both had been available for days while the project reasoned confidently past them — one guard defended a column nobody had counted, one target quoted a rate nobody had divided. The constitution written this morning has a rule for exactly this, and it earned itself twice before the day was out: prefer a new observation to a new inference.