the ledger notes
Three silent gaps, three cheap looks
2026-08-14. Launch week, working on the capture pipeline — the part that fetches each domain's AI-discovery files (llms.txt, robots.txt, sitemaps, homepage JSON-LD) so a report can show the version we saw at collection time instead of re-fetching the site live. Three things went wrong this day. None of them announced itself. Each was caught by looking, not by the code telling us. That's the whole post.
1. The backfill that would have stored nothing
The plan was clean: 100 workers across two Macs, 9,057 domains, capture every present file's body so reports serve the snapshot. The fetcher already had a guard — it stored a body only when the content hash changed, so routine re-checks didn't churn storage for identical bytes. Sensible.
It also would have made the entire backfill a no-op.
Every one of the 19,790 existing present files was fetched before body-storage existed: hash already set, body NULL. On re-fetch the hash still matches → "unchanged" → skip. The crawl would have fetched all 9,057 domains, spun 100 workers for fifteen minutes, and stored zero bodies. A green run, a drained queue, nothing to show for it.
It got caught because the ask was "make sure the rows are in the db to ingest the data." One query: 19,790 present, 0 with bodies. A single live fetch printed changed=false while the body it was holding went nowhere. The fix was three lines — store the body when the row is "unchanged but bodyless," not just when the content changed. Then it worked: 17,512 bodies, 0 present-but-bodyless.
The lesson has teeth: an idempotency guard and a backfill are natural enemies. The exact condition that keeps re-runs cheap — "skip if unchanged" — is the condition that makes the first run a no-op for everything that predates the feature. Nothing in the code was wrong. It was wrong about history.
2. Six percent of the captures were filed under a name nobody searches
Building the serving layer, the first real domain I tested came back empty. Not a bug in the new code — a data bug the new code finally exposed. The report resolves /d/{host} by the bare hostname (no www.), but 1,048 captured bodies (6%) were filed under www.-prefixed domain rows. 437 malformed rows in all, orphaning 3,059 file rows: captured correctly, complete, and unreachable, because they were filed under www.ford.com while every lookup asks for ford.com.
Then the second turn, inside the first. I went to plug the leak — and found it already closed. Every one of the 437 rows was dated Aug 6–9; zero since. The bare_host virtual column added days earlier for an unrelated slow-query fix had incidentally cleaned the one write path that leaked (Domain::upsert(), now grouping by bare_host). I'd set out to stop an active leak and found a quiet one. The real work was cleanup — 228 rows renamed to bare, 209 redundant collisions deleted, 18 unique captures rescued (sites where www. was the canonical host and held a body the bare twin lacked) — plus a backstop so it can't reopen.
The backstop is the point. The leak path was upsert, which writes straight past every Eloquent mutator, cast, and event. An app-level "normalize the hostname" guard was structurally incomplete — it could never cover the one path that skips the app. The only complete guard is at the DB: CHECK (hostname NOT LIKE 'www.%'). It fails loud (SQLSTATE 23000) on any future regression, upsert included. Verified by trying to break it on prod, not by watching it deploy.
3. The outage was the alarm
A page landed: "airank-collector stuck at 0/2 after heals — needs a look." Reflex says outage.
The logs said otherwise: logged_in · Sessions to run: 0 · Nothing to collect — every matched phrase is already at target · SIGTERM received during idle wait, exiting. The collector is a run-to-idle batch: when there's no work, 0/N replicas is the healthy state. And the healer — swarm-heal.sh, cron every 5 minutes — force-restarts anything below its replica count. So it was the SIGTERM source that kept killing the idle container, and, after four strikes, the pager that complained the container wouldn't stay up. A closed loop: heal → kill the idle container → still 0/2 → heal → page.
Collection was fine the entire time: 5,673 observations in the last 24h, newest three minutes old. The fix was one line — skip airank-collector* in the heal loop, exactly like the stateful db_* services were already skipped. A monitor that can't tell idle-by-design from down will fight the service and then page you about the fight.
One shape
Three failures, one silhouette. None surfaced on its own: a green deploy, a running service, a fetch that returned 200. Each needed a deliberate look to exist at all — a row count, a test against a real domain, four lines of a log. The code was confidently, quietly wrong in all three, and confidence is precisely the thing that does not survive a measurement. The cheapest query I ran all day — COUNT(*) WHERE body IS NULL — saved the most expensive waste: a hundred workers moving nothing.