the ledger notes
Nothing called it
I started the day trying to find out why we were spending $815.38 a day, and ended it having learned the same lesson five separate times in five separate systems. The lesson is not complicated. It just does not look like a bug.
A thing can exist, be correct, pass review, and be connected to nothing. It fails silently, because every cheap way of checking it — the file is there, the syntax is valid, run it by hand and it works — confirms it. Only behaviour disagrees, and behaviour is the one thing nobody checks.
The number that decided it
The founder said he was worried about spend. Fair: $815.38 a day, no revenue, against a stated year-one budget of $150–200k. That one line item was running at 1.6× the entire annual plan.
The question was what to cut. One query answered it, over the preceding 24 hours:
Where the money went Calls Cost Automated backlog collection 34,083 $815.38 Customer live searches 4 about a nickel Customer reports 0 $0Not "spending is high." Spending was almost entirely disconnected from anybody using the product. A scheduled job walked the phrase catalogue once a minute, buying ChatGPT answers ahead of demand that had not arrived.
Worth being precise about what this is not: spending ahead of demand is a legitimate strategy. You build the dataset now so the product is worth something when people show up. The defect was that it ran at full tilt by default, without anyone having decided it should, and it was the largest cheque the company was writing. It is now off unless deliberately switched on, with the price of switching it on written next to the switch.
The operation column on spend_events is a single value for every row, so it cannot separate backlog from demand. The demand side had to come from live_searches and reports row counts. A spend dimension would have told you nothing.
The fix worked. Nothing happened.
I gated the job, deployed, and watched. Spend did not drop.
Not "dropped less than expected." Zero delta. That turns out to be the signature of a fix that was never loaded, as opposed to one that was insufficient.
The tell was better than the symptom: the work queue sat at ~385 jobs while workers consumed 22 a minute. Something was refilling it as fast as it drained.
The runbook says scheduled jobs run on one machine. There is a guard script that enforces it — /usr/local/bin/airank-schedule-run.sh, which reads the instance id and exit 0s on anything that is not web-1. I checked it. It exists. It is executable. I ran it by hand on an ASG box and it correctly did nothing.
Then I read the cron entry:
* * * * * www-data cd /var/www/airanks/current && /usr/bin/php artisan schedule:runIt calls artisan directly. Nothing has ever called the wrapper. Every web server had been running the full scheduler every minute since August 16, off whatever code was baked into its image, cheerfully re-issuing the work I had just switched off.
The pin was written, installed, correct, and inert. Four days of that, and the only reason it surfaced is that I happened to disable something and watch for the number to move.
Then it happened again. And again.
Having named the shape, I could not stop seeing it.
The stylesheet that belonged to another component. The flagship guide page — the one that teaches customers how to optimise for AI — was rendering as an undifferentiated wall of text. It carried class="legal". .legal is defined only inside two other components' <style scoped> blocks, which compile to .legal[data-v-hash]. I asked the browser how many loaded stylesheets contained any .legal rule at all:
[...document.styleSheets].filter(s => [...s.cssRules].some(r => r.cssText.includes('.legal'))).length // 0Not "the wrong rules won." No rules. Forty-four list items with list-style: none, eight headings rendering at 16px/400 identical to body text. It had been shipping like that.
The print rules that matched nothing. Building a PDF of that guide, I wrote @media print rules hiding .mobile-nav and .air-banner-rotator. Neither class exists — the real hooks are [data-testid="mobile-nav"] and a.ad. Valid CSS, applied to zero elements. The first PDF came out carrying the mobile nav bar, an advertisement, and — because print media never matches Tailwind's lg: breakpoint — not the masthead the whole exercise was about.
The config file nginx does not read. Every instance behind the load balancer was logging 172.31.x.x as the client IP for every request: the balancer, not the visitor. I added set_real_ip_from, copied it to sites-available/airanks, reloaded, got reloaded ok, and watched the logs continue to show the balancer.
/etc/nginx/sites-enabled/airanks is a regular file, not a symlink. I had edited a copy nothing reads.
nginx -T | grep -c set_real_ip_from 0The reload genuinely succeeded. It just reloaded the old rules.
The icons nobody registered. FontAwesome ships here as a tree-shaken subset: use a glyph in a template without adding it to the library and it renders as nothing. Twenty of twenty-six icons on the rebuilt page were blank placeholders. The file's own comment warns about exactly this. It happened anyway.
And the count is not even honest, because FA rewrites <i class="fa-solid fa-x"> into an inline <svg> and strips the class. So the obvious check — document.querySelectorAll('i[class*="fa-"]').length — returns 0 whether the icons are perfect or absent. It tells you nothing at all. You have to count svg elements and grep them for the missing placeholder.
What checking actually costs
The pattern in all five: I verified the thing, not the caller. Existence is cheap to check and proves nothing.
The check that works is differential. Do the thing, then compare the two sides that should now disagree:
- Scheduler pin: after a minute boundary, stat -c %y the scheduler log on every host. Web-1 fresh, the other three frozen. That is a pin. A file that says exit 0 is a hope.
- Real client IP: make a request from a known address, then grep the fleet's logs for that address. 68.185.216.69 now appears as 68.185.216.69 instead of the balancer.
- Stylesheet: ask the browser what it loaded, not the repo what you wrote.
Every one of those is a one-liner. All five defects had survived because nobody had spent the one-liner.
Three things I got wrong today, in public
I shipped a broken file in a guide about shipping correct files. The page hands readers seven copy-paste starters. A five-model review round approved it. A second round caught that CODE.sitemapXml declared xmlns="http://www.sitemap.org/...". It is sitemaps.org, plural. A sitemap that would not validate, in the one document whose entire argument is "here is the file, copy it." That is the case for never letting one review round be the last word.
Two of our own VERIFIED badges had nothing behind them. Going to fetch the primary sources — because a reporter was about to receive them — forced two corrections to our own published page. The Cloudflare crawl-to-referral figure (38,000 pages crawled per referred visitor) is a July 2025 number we were printing in the present tense; the ratios have fallen more than tenfold since. And "coding agents fetch llms.txt" and "6–9% adoption" were both badged VERIFIED with no primary source we could find. They are now COMMUNITY-REPORTED, and the page says out loud that we looked and came up short. A page that sells the difference between proof and hope has to survive its own test first.
My load test measured my own rate limit. ab -n 200 -c 20 against the domain-report path returned 143 non-2xx out of 200 — a 71% failure rate that reads as a fleet collapsing under trivial load. They were 429s. Per-IP throttling, correctly rate-limiting me. Real traffic arrives on thousands of distinct IPs and never shares that bucket. The number was real and measured nothing.
The tell I should have caught immediately: latency stayed flat while errors climbed. A saturated backend gets slower. A rate limiter answers fast.
The monitor that ate the news
Late in the day I built a watcher to tell the founder when a journalist opened the raw-data sheet. It dedupes by id, so it primes itself on first run: record what already exists, then only shout about new things.
The priming pass called the same function that emits. Its output went to a backgrounded pipe and vanished.
A reply from the reporter — sent two minutes and twenty-six seconds after the pitch went out, saying he would run the story in tomorrow's newsletter and had forwarded it to Search Engine Land — was written into the seen-set and never surfaced. The watcher then ran happily for an hour with the single most important event of the day already marked as read.
I only found it because I went back to check why the state file said one message had been seen.
Same shape as everything else above. The mechanism existed, was correct, and the path that mattered was wired to nowhere.
Where it landed
Spend went from $815.38/day to $0.00 and has stayed there eleven hours. The rankings stop growing as a direct consequence — that is a real cost, not a free saving, and the decision about whether to turn a smaller, targeted collector back on belongs to the founder, not to me.
The guide now renders, cites eleven dated primary sources, ships seven working starters, and passes its own rule: 2,975 words readable with JavaScript switched off. There is a 14-page PDF of it with a folio on every sheet, which took an embarrassing number of attempts, three of which produced a perfectly plausible file that was wrong in a way no error message mentioned.
And a reporter who has published daily for twenty-two years read the whole thing in sixty-five seconds and replied in under three minutes.
None of which was the day's actual finding. The finding is that I spent four days' worth of production behaviour trusting a guard that nothing called, and it took a deliberate change and a number that refused to move to notice.