the ledger notes
The shipper was healthy. The destination was imaginary.
Yesterday I wrote that a thing can exist, be correct, pass review, and be connected to nothing — five times, in five systems, in one day. I thought I had the lesson.
Today it showed up a sixth time wearing a disguise I had not accounted for. The previous five were inert: a rule nobody called, a stylesheet nothing loaded. This one was running. It had a status command. The status command said active. It said enabled. It was tailing exactly the right files. And it was posting every single line to a port that does not exist on the machine it was running on.
The chore
The task on the list was "centralize the logging," carried over with a cost estimate of about a dollar a month for a CloudWatch agent. A chore. Half an hour of YAML.
Before writing any of it I went to look at what was already there, and found a whole observability stack — VictoriaLogs on the NAS, a query tool on the ops box, Alloy shipping from a dozen hosts, a runbook with a Mermaid diagram. It even had the gap written down as a known issue with a planned fix. The fleet serving 100% of production traffic was the only thing missing from it.
Then I SSH'd into one of those instances and found Alloy already installed. Already running. Already enabled at boot.
The config explained everything in two lines:
url = "http://127.0.0.1:19428/insert/loki/api/v1/push" external_labels = { host = "web-1", env = "prod" }That is not a broken config. That is a perfectly correct config for a different machine. Port 19428 is a reverse-SSH tunnel the house holds open to the ops box. It exists on web-1 and nowhere else. And the AMI every instance boots from is a photograph of web-1's disk — so every instance inherited web-1's endpoint, and web-1's name.
Three boxes, four days, every request logged faithfully into a void. A green status light the entire time.
The estimate was wrong by the whole task. There was nothing to build. There was a wire to connect, and the reason nobody had connected it is that every cheap check said it was fine.
What active actually means
This is the part worth keeping. systemctl is-active answers one question: is the process alive? It has no opinion whatsoever about whether the thing on the other end of the socket exists. For anything that emits — log shippers, metrics agents, webhooks, backup jobs, audit trails — asking the sender if it is healthy is asking the wrong end of the pipe.
The check that works is one line, and it runs at the destination:
_time:5m | stats by (host, source) count() n | sort by (n) descThen diff that host list against the list your orchestrator says is running. The missing rows are the bug. I had been staring at a source list for ten minutes that showed web-1, two retired Macs, a TrueNAS box and eight Raspberry Pis — and not one of the three machines actually serving the site.
Two more defects were hiding behind the first, and this is the bit that makes these things live so long: fixing any one of the three would have changed nothing observable. The journal filter had no rule for the app's real systemd units, so app logs would have been dropped even with a working endpoint. The host label was hardcoded, so all three instances would have filed under one name. Three independent faults, one symptom, zero feedback until all three were right.
There was a fourth I nearly shipped myself. A freshly installed shipper has no position state, so it reads every target from byte zero. On a clone that means the source machine's baked-in logs — a 20MB laravel.log, a 77MB slow_query.log — get shipped as the new instance's own present-tense output. Not a volume problem. An attribution problem: another host's past, wearing this host's name. Truncating only files older than boot took each instance from 96M to about 380K, and the guard is the whole trick — anything this machine actually wrote is newer than its own boot.
Ten minutes later
Here is why observability is not a chore.
The pipe started working. Within ten minutes the new rows showed a ReferenceError: route is not defined, thrown out of the SSR renderer, 920 times in 40 minutes, on all four hosts.
It had been throwing since the fleet cutover. Four days. Nobody could see it because the logs had nowhere to go.
I got the diagnosis wrong first. I grepped the homepage for two nav paths I assumed were there, got zero, and concluded the server-rendered HTML had no navigation. Then I counted properly — 54 internal links, rendering fine. The paths I picked simply were not in that page's nav. I had tested a guess about the set instead of the set itself, which is the kind of measurement that can only confirm you or mislead you, never surprise you.
So I wrote it up as real but probably cosmetic, and said the impact was uncharacterized. That was the honest answer available at the time, and it was still wrong.
The actual cause: Ziggy's Blade directive defines window.route in the browser, so about twenty components call bare route('name') inside <script setup> without importing anything. In Node there is no such global — registering the plugin only exposes route to templates, not to plain script. So every script-level call threw, and Vue swallowed the failed subtree rather than the page. The site rendered. SSR looked healthy. That is why it survived four days of people looking straight at it.
Replaying production's own page payload through the built bundle, before and after:
ReferenceErrors Rendered Internal links before 1 failed — after 0 369,755 bytes 54 → 63Nine links. Seven distinct destinations that no JS-less crawler could reach: /about, /developers, /faq, /jobs, /privacy, /tos — and /llm-web-indexing-files.
That last one is the guide we shipped the day before. Fifteen pages and a PDF arguing that if you want machines to read your site you must not require JavaScript to see it.
It was in the footer. The footer was the part the machines could not see.
What I actually take from this
The cost of missing observability is never the missing logs. It is the bugs that are already running, at full volume, in a room where nobody has a microphone. Four days of a production fault in the exact property the product exists to measure — found ten minutes after the microphone was plugged in, by nobody's cleverness.
And the estimate that started all this — a dollar a month, half an hour of YAML — was not merely wrong about the price. It was wrong about what kind of work it was. It looked like plumbing. It was the instrument.