Three alarms, none of them wired

This morning I wrote about five hours of dead collector and nobody being paged. I spent the rest of the day building the thing that should have paged me.

Then I went looking for why the existing alarms hadn't. The answer turned out to be the same answer three times, and it is not "we hadn't built them yet." We had built them. They were tested. They were documented in the README. Two of them had been committed within the previous twenty-four hours, with careful follow-up commits tuning their edge cases.

None of them were plugged into anything.

The health check that couldn't see anything

I started with a status question — where is this project actually at — and ran the health command.

Database: UNREACHABLE - SQLSTATE[HY000] [1129] Host '192.168.1.33' is blocked because of many connection errors; unblock with 'mariadb-admin flush-hosts'

Every section below it degraded at once. Workers: unqueryable. Collection recency: unqueryable. One blocked host and the entire observability surface went dark together.

My first instinct was that something was hammering the database with bad credentials. It wasn't. max_connect_errors was at its default of 100, and skip_name_resolve was OFF — which is also the default, and which means MariaDB performs a reverse-DNS lookup on the client IP for every single connection. A LAN with no PTR records fails those lookups, and every failed lookup counts toward the limit. A busy app host burns through 100 in ordinary operation and then locks itself out of its own database.

Nothing was wrong with the credentials. Nothing was wrong with the network. The security feature was eating the application.

The tempting fix is max_connect_errors = 100000, and I did set that, but it's a bigger bucket under the same leak. The actual fix is skip_name_resolve = ON, which removes the error source and drops connect latency as a side effect.

That one has a trap in it, which is why I checked before restarting: with name resolution off, any grant written against a hostname stops resolving and that user can never connect again. If I'd restarted first and looked second, I'd have turned a recoverable annoyance into a lockout.

SELECT user, host FROM mysql.user ORDER BY user;

Every grant was IP-based or localhost. Safe. Restart, verify, move on.

The detector that shipped switched off

With the database visible again, the collection-recency check finally ran:

newest observation: 5.3h ago staleness threshold not configured (AIR_CHATGPT_STALE_AFTER_HOURS) - informational only

Collection had been dark for over five hours, and the feature built specifically to notice that printed a note.

AIR_CHATGPT_STALE_AFTER_HOURS defaulted to null. It was commented out in .env.example, absent from .env, absent from stack.yml. The staleness branch had never executed once in production.

And there were two commits from that same morning tuning its behaviour. One of them fixed an off-by-one where a "stale after 1h" threshold first fired at 2h. Real bug, correct fix, careful comment explaining the float-vs-int reasoning — on a code path that had never run.

That's the part I want to keep. Not "we forgot to set a variable." We maintained it. Attentively. Careful maintenance of an unarmed detector looks exactly like careful maintenance of a working one, which is why it survived.

The reason it shipped disabled was written down and, in isolation, sound: the collector is deliberately scaled to zero much of the time, so an unconditional staleness failure would cry wolf. Conservatism. Except conservatism that results in never firing isn't conservative, it's off.

The one that made it a pattern instead of a bug

Two for two made me suspicious, so I went looking at the third check on the same screen — the worker table:

| mbp.shoemoney.ai | online | 16 hours ago | 177371 | 1231 | 0.7% | | hueb.shoemoney.ai | online | 18 hours ago | 1 | 0 | 0.0% |

Online. Sixteen hours since last contact.

The table printed a stored status column. Four methods wrote 'online' to it. One method wrote it back to 'offline' — reapStale(), whose docblock read "Mark silent workers offline. Run from the scheduler."

$ grep -rn "reapStale" . --include=*.php | grep -v vendor/ app/Services/WorkerRegistry.php:74: public function reapStale(): int

One hit. Its own definition. There is no scheduler in this repo. There never was. Every worker that ever booted had been reading online forever.

Three shapes of the same failure, found in about an hour:

The mechanism Why it never fired Staleness correct, tested env var set in no environment Worker liveness correct, tested method with zero callers Everything else correct, tested exit code discarded by || true

That third row is the one I nearly missed. Both container healthchecks run the health command like this:

OUT=$(php artisan air:health 2>&1) || true echo "$OUT" | grep -q "Database: reachable" && echo "$OUT" | grep -q "Redis: reachable"

The || true throws away the exit code carrying every verdict the command computes. And that was deliberate — there's a comment explaining it, and the reasoning is good: queue backlog and Horizon state are fleet-wide facts, not this container's health, so a paused Horizon shouldn't mark every worker unhealthy.

So arming the staleness threshold would have accomplished nothing. The verdict would have been computed correctly and then dropped on the floor. I'd have "fixed" it, watched the tests pass, and shipped a detector that still paged nobody.

The fix that resolved the argument instead of picking a side

The cry-wolf problem and the never-fires problem looked like a tradeoff: arm it and get noise when the collector is intentionally off, or leave it disarmed and get silence when it breaks.

It isn't a tradeoff. It's a misplaced check.

Put the staleness gate in the collector's own healthcheck, and the dilemma evaporates:

Collector deliberately off → no container → the check doesn't run → silence, by construction.
Collector running while the corpus goes stale → unambiguously broken → unhealthy.

No env var courage required, because the thing that makes the alarm appropriate is the same thing that makes the container exist. When a gate needs a knob to stay armed, the gate is usually in the wrong place.

One trap on the way in: the collector might legitimately start into an already-stale corpus after a long deliberate shutdown, and would then fail its own healthcheck while trying to fix the very condition it was failing on. A restart loop, on exactly the operation the check exists to protect. start_period went from 90s to 600s to give a cold start room to land a row.

Two probes that lied to me on the way

The exit code that wasn't. Early on I ran the health command piped through tail, appended echo "EXIT: $?", and got 0 — with the database unreachable. I got about ten seconds into drafting "the health check fails open on total database failure" before realizing $? after a pipe is tail's exit code. handle() ANDs the database verdict correctly. It was always fine. I nearly reported a fabricated bug about a fail-open in the middle of an investigation about fail-opens.

The process that was watching itself. Later, waiting on an ARM image build on a Pi:

ssh $HOST 'pgrep -f "buildx build" >/dev/null && echo "BUILD STILL RUNNING"'

It said RUNNING long after the log printed pushing manifest ... DONE 91.4s and the tag was live in the registry. pgrep -f matches full command lines, and sshd was at that moment running a command line containing the literal string buildx build. The probe was matching itself. Wrapped in an until loop it would have waited forever for a condition that could never be true.

Both are the same mistake wearing different clothes: I measured the thing that was easy to measure instead of the thing I actually wanted to know.

The fix wasn't live, and neither was the commit

Tests green, mutation-tested, committed, pushed. I nearly called it done.

Then I noticed the Horizon plist referenced ~/Sites/airank, which doesn't exist on my laptop. It exists on hueb. So does ~/Projects/airank. Two checkouts on the deploy host, plus mine, and the live process was using a third combination of the two: it ran from hueb's ~/Projects/airank, sitting 30+ commits behind main, while the path the deploy config named was a different stale clone.

The only trustworthy answer came from asking the process itself:

ssh 192.168.1.3 'lsof -p <pid> | awk "\$4==\"cwd\"{print \$NF}"' /Users/shoemoney/Projects/airank

A correct, tested, pushed fix had changed nothing about the running system, and would have kept changing nothing indefinitely, silently, while I believed otherwise. Which is precisely the failure class I'd spent the morning documenting — arriving from a completely different direction.

Same for the container. The healthcheck gate lives inside an image; the service was still running the previous tag. A commit is not a deploy. So: build on the Pi (local Docker was down), push, and then the step that actually matters —

docker run --rm --entrypoint sh <image:tag> -c 'grep -c "STALE - older than" /path/healthcheck.sh'

Prove the fix is inside the artifact, not inside the commit the artifact was supposedly built from.

I also made a mess doing it. My rsync -a excluded .git, node_modules, and vendor — three directories — and copied .env straight to the Pi at mode 644. I'd reasoned about .dockerignore and concluded secrets were safe, which was true and irrelevant: .dockerignore governs the build, not the transfer. Two different exclusion lists, and I consulted the wrong one.

Then I turned it on, and the question changed

With everything armed, I started the collector. It came up healthy and began producing. The log showed phrase ids stepping 27, 28 — one at a time, which is the absence of an earlier sharding bug whose signature was ids jumping 105, 110, 115.

All good. Then I looked at what it was actually producing.

Sessions to run: 13 OK phrase 27 (logged_out) {"skipped":true,"phraseId":28,"reason":"products_implausible:no_recognized_brand",...}

Total corpus: 60 observations. Against 1,463 phrases.

Most sessions get skipped at products_implausible:no_recognized_brand — after a full ~35 second ChatGPT session has already run. The expensive part completes and the result is discarded at the brand-recognition gate.

I had spent the day on monitoring and deployment for a collection pipeline whose problem is not collection. The pacing math, the replica count, the proxy exits, the shard arithmetic — all of that governs how many sessions we can run. None of it governs how many produce data. Scaling to five replicas would buy five times the sessions and five times the skips.

The bottleneck was never throughput. It's the gazetteer.

That reframes the next move entirely. I came in asking "how do we collect more" and I'm leaving with "why are we throwing most of it away."

What today cost and what it bought

Belief Killed by Cost The DB block is a credentials/network problem SHOW GLOBAL VARIABLES LIKE 'skip_name_resolve' ~30 seconds The staleness detector works, it just wasn't set reading who consumes its exit code ~5 minutes reapStale() runs from the scheduler one grep ~10 seconds The fix is live because it's pushed lsof on the running process ~2 minutes Collection is throughput-bound reading the collector's own log for 60 seconds ~1 minute

Every one of those was cheaper than the work it prevented, and the last one was cheapest and worth the most.

The rule I'm taking out of today: never conclude a health signal works because the code and the tests exist. Trace it to a consumer that can actually fail something or page someone. Three questions, all cheap:

What reads this exit code?
What calls this method?
Is this variable set in every deploy surface, not just .env.example?

A detector nobody reads is not a conservative detector. It's decoration with test coverage.

AIRANKS — The Authoritative Rankings for AI Web Content