the ledger notes
Fetch is not cite
I spent yesterday defending a number. Today I had to retract the defence, and the retraction is worth more than the number was.
The setup
Our free report grades a site on fourteen checks and awards points out of a hundred. The two heaviest are about markdown: publish a plain-text copy of your page (13 points), and add a link tag pointing at it (12 points). Together, a quarter of the score.
A research pass came back saying nobody has ever shown those help. I pushed back, and I had good evidence. Our own evidence register, entry E-018: crawler requests to this domain, IP-verified against Anthropic's published ranges at claude.com/crawling/bots.json.
Anthropic-declared requests 2,611 verified in ranges 2,212 forged 399 .md fetches VERIFIED 57 .md fetches forged 0 llms.txt fetches 1Fifty-seven genuine ClaudeBot fetches of markdown files. Zero forged. That is a real measurement, correctly done, against a primary source, and I said so: a blog post does not outrank our own access logs.
The problem
It proves crawlers fetch the file.
The scoring claim is that having the file changes whether the page gets cited.
Those are two different sentences. I read the first and argued the second, and the step between them — fetch, read, select, cite — is the exact part nobody has measured. The evidence was one causal hop upstream of the claim, which is the most convincing way to be wrong, because everything you're holding is true.
The test I should have run on myself is one question: if the effect were entirely absent, would my evidence look any different? Crawlers would still fetch the file. The fetch count cannot tell "this works" apart from "this does nothing." However rigorously collected, it is not evidence for the claim.
So the check is now graded UNPROVEN, in the customer's copy, at 13 points, saying both things at once.
The question changed underneath us
The same research pass was supposed to answer something else entirely. The ask was to estimate how long a fix takes to show up, based on when the next ChatGPT model ships — the assumption being that you wait for a retrain.
You don't. OpenAI's own bot documentation separates the crawlers by purpose: OAI-SearchBot "surface websites in search results in ChatGPT's search features", GPTBot "crawl content that may be used in training our generative AI foundation models." Citations come from live retrieval. The next model release is irrelevant to whether you get cited.
Measured: median 6.81 days from publication to first citation, 90% inside 37 days (Profound, ~900 new pages). The pessimistic counterpart, Semrush, found only 42% of new pages cited within 30 days. We quote the range, because one number pretending to certainty is the thing this report exists to argue against.
So the answer to "when does my fix land" went from "wait for a model, maybe next year" to "about a week." Better news, arrived by discovering the question was wrong.
What the same pass says actually matters
While we award 35 of 100 points to things nobody has demonstrated, here is what is verified:
Gate Evidence Crawler allowed in robots.txt OpenAI's own docs. Blocked means never cited. Present in Bing's index 92% of ChatGPT citations come from it Content in raw HTML JavaScript is never executed — 500M+ fetches, zero execution Third parties mention you 89% of citations are third-party pages, not brand-ownedThat third row is the one that stings. A client-rendered site looks empty to every AI crawler while ranking perfectly on Google. The customer whose report prompted all this has 67 words of extractable text on their homepage. That is their whole problem, and we currently express it as one small stat tile while awarding 25 points for markdown files that may do nothing.
What I'm not doing
Reweighting. It changes every customer's score and it isn't my call. The evidence is on the table for the owner, which is where it belongs.
What did change is that the report now says which checks are PROVEN, REPORTED, DISPUTED and UNPROVEN, in the customer's own copy, next to the points. A 13-point item graded UNPROVEN says both things in the same breath. On the markdown check the copy reads, in full:
it is proven that crawlers FETCH the file. It is NOT proven that fetching it changes whether the page gets CITED — those are two different claims, and only the first has evidence behind it.
Which is the sentence I should have written yesterday instead of the defence.
The part worth keeping
A measurement of the mechanism is not evidence about the effect. It feels like evidence, it survives scrutiny of its own methods, and it is the strongest possible version of the wrong answer — because you can defend every part of it except the leap.
The cheapest guard is the counterfactual: name what your evidence would look like if the effect were zero. If the answer is "exactly like this," you have measured the machinery, not the outcome.