the ledger notes
Three Numbers We Made Up
title: "Three numbers we made up, and the tests that watched" date: 2026-08-24
We shipped a lead-capture funnel this week. Someone lands on airanks.net, types their domain and a keyword into a popup, gives us an email, and we go find out whether ChatGPT cites them.
The engineering was fine. Fourteen hundred tests, green. What follows is about the three numbers that were wrong anyway, and the fact that every one of them was created by the fix for the one before it.
One
A domain's report shows an AIR Score out of 100. We compute those nightly and store them.
Ashton Woods, a national homebuilder, has a row in that table reading air_score = 0, percentile = NULL, occurrences = 0, phrases_count = 0. A score of zero, computed over nothing at all. The row exists because someone once looked the hostname up, not because anyone measured it.
The report gated on "does a row exist". So it would have printed, to a real customer:
Your AIR Score: 0
That is not a low score. That is us never having looked, formatted as a measurement. About 1,225 of 9,982 rows in that table carry a score backed by zero observed phrases, so it was not a one-off.
Fixed: derive the score's availability from the evidence, not from the row. Regression test uses the exact Ashton Woods shape.
Two
The fix worked, and produced this, in the rendered PDF, four lines apart:
0 Citations across 0 tracked phrases
This Domain's Most-Cited Pages — 1 citation, https://www.ashtonwoods.com/
The header came from the stale cached row. The panel came from a live query. Both from the same function, disagreeing with each other on the same page.
Nothing caught it. It was visible only by rendering the document and reading it.
Fixed: derive the header from the very lists printed beneath it, so it cannot contradict the body.
While making that change I moved the score's provenance gate onto the live counts, which re-admitted bug one within about ninety seconds of removing it. Caught on re-reading my own diff. The live count is 1 and the row's is 0; they are not interchangeable, and there is now a comment saying so where the next person will find it.
Three
Then an adversarial review read all 4,700 lines of the branch at once, which nobody had done, and found this:
$phrasesCount = count($citedFor); // citedFor() -> limit(10) $occurrences = array_sum(array_column($topCitedPages, 'citations')); // -> limit(10)Those two feed the header "{occurrences} Citations across {phrases_count} tracked phrases", printed under a note reading "Across our whole tracked dataset".
They are counts of lists capped at ten.
I ran the real query against the busiest host in our data, tomsguide.com, 123,908 rows:
phrases = 993 occurrences = 99,244 pages = 1,349The shipped code would have rendered that as "10 Citations across 10 tracked phrases". Four orders of magnitude, stated as a measurement, beside a claim that it covered everything we track.
I wrote that one. It was the fix for bug two.
Three instances, one shape: a displayed figure and its neighbouring evidence coming from sources with different freshness. Each fix moved the error one layer outward rather than removing it. Cached row versus live query. Then live query versus truncated live query. The third was mine and I would not have found it, because I had just convinced myself the area was fixed.
The test was in the room every time
The code for bug three had a test. It seeded twelve phrases and twelve pages against a cap of ten — precisely the boundary case — and then asserted:
assertLessThanOrEqual(self::CITED_FOR_LIMIT, count($profile['cited_for']))It never touched either header field. The reviewer's sentence is the one I keep coming back to: the test builds the bug and looks the other way.
That is worse than no test. A missing test is a known gap. This one made the area look covered, which is exactly why I stopped looking at it.
While tightening a different guard I managed the mirror-image error. Mutation-testing the rule that no paid work happens before email verification, I injected a dispatch to prove the test could fail. It failed — with Attempt to read property "id" on null, because I had put the injection where the variable was out of scope. Red, for a reason with nothing to do with the property. Proof of nothing. Redone properly it read:
The following jobs were dispatched unexpectedly: App\Jobs\BuildReportJobRed is not proof. Red for the right reason is.
The two emails nobody had read
The funnel's promise is: give us your email, confirm it, and we send you the report. Two emails carry that. Neither had ever been looked at.
The first, rendered from the real notification class, was Laravel's stock verification mail:
Subject: Verify your email address Please click the button below to verify your email address. If you did not create an account, no further action is required.
Ninety seconds earlier this person typed their domain into a popup because they wanted a report. They do not believe they created an account. That closing line, which ships in every Laravel app on earth, tells them the one click that starts all the work is safe to ignore.
The second fired when the report finished. The payload said your_citations: 0 across five sampled answers. The email said:
Overall score: 84 / 100
and linked out. Someone skimming on a phone reads a passing grade and closes it, never learning the single fact they gave us their address for. It was a link, not a payoff. It now leads with "ChatGPT cited coldbrand.test 0 of 5 times."
Both emails passed their tests, because a test asserts that mail was sent, not that it was worth sending.
What I was guarding, versus what kills you
I spent real effort on fetch amplification: an attacker naming a victim's domain so our servers hammer it. A seven-model pre-mortem ranked something else first, and was right.
Bots found the submit endpoint in week two, every submit mailed a stranger, and SES put the whole domain under review while the new webhook dutifully recorded each bounce, because recording is not suppression.
We had just built SES bounce handling. It stops us re-mailing an address we already know is bad. The funnel's entire exposure is first sends to novel addresses, which is precisely the send suppression cannot prevent, and bounce rate is what AWS scores. There was no bot gate at all.
That is not a bug in the suppression list. It is the suppression list solving the adjacent problem convincingly enough that I stopped asking.
The through-line
None of this was found by tests. The tests were green throughout, and in two places they were actively part of the problem.
It was found by rendering the PDF and reading it, by opening the email, by driving the funnel in a browser, by running the query against the real table, and by having something read the whole diff at once instead of one file at a time.
The cheapest of those was the query. Sixteen hundred milliseconds to learn that a number in our customer-facing report was wrong by four orders of magnitude.
We sell a product whose entire claim is that we do not make numbers up. In one week, on one branch, we made up three. Every one was caught by looking at the thing a customer would see, and none by the fourteen hundred tests that said we were fine.