Four hundred and seven sessions, no evidence

The collector ran 407 browser sessions in twenty-five minutes and produced nothing. No data, no saved pages, and — this is the part that mattered — no record that any of it had happened.

I spent the next several hours being confidently wrong about why, three separate times. Then a single line of captured HTML answered it. The gap between those two facts is the whole post.

The bug that hid the bug

Every failed session appended a line like this to the log:

{"skipped":true,"phraseId":42,"reason":"navigation_failed","htmlCaptured":false}

htmlCaptured: false, 407 times out of 407. And zero rows in the database. The relevant code:

// A capture with no HTML is nothing to archive - the page was gone before we could // read it. Skip rather than storing an empty object that looks like evidence. if (empty($row['html'])) { continue; } Storage::disk('minio')->put($objectKey, gzencode($row['html'], 9)); ChatgptCapture::create([...]); // never reached

The comment is half right, which is exactly why it survived review. An empty object in storage genuinely would look like evidence. But continue skips the next statement too — and that one writes the ledger row, the record that an attempt happened at all.

Two things break at once, and both are silent.

The retry logic counts attempts from those rows. No row, no attempt, so the phrase stays eligible and gets picked up again next batch. Forever. In the logs I found phrase 92 attempted six times in twenty-five minutes, along with 64, 112 and 100. Every one of those attempts cost a real browser session against a rate-limited third party. The system looked busy and was in a loop.

The second breakage is worse and less obvious: with no artifact and no row, the failure has no evidence, so every explanation is a hypothesis. That's how I lost the evening.

Three confident wrong answers

Wrong answer one: the browser profiles are colliding. The running image predated a fix that gives each replica its own Chromium profile directory, and it was running four replicas per host. Four browsers, one profile, Chromium's SingletonLock — a genuinely good theory. I had been repeating it for hours as the leading suspect.

I deployed the isolation fix and ran at one replica per node, where profile sharing is impossible by construction. Still ~100% failure. Dead.

Wrong answer two: OpenAI is rate-limiting us for going 20-wide. Also plausible; there was prior history of exactly that, and a 78% navigation-failure rate is what being throttled looks like.

Dead too, and more embarrassingly: traffic was never reaching OpenAI at all.

Wrong answer three: the proxy credentials aren't in the container. I exec'd in, checked the environment, and found AIR_PROXY_USERNAME and AIR_PROXY_PASSWORD empty. That felt like the answer. I nearly wrote it up.

It was a broken probe. docker exec starts a new process and does not run the ENTRYPOINT — and the entrypoint is precisely what loads /run/secrets/* into environment variables. The credentials were correct the whole time; my measurement couldn't see them. If I'd shipped that diagnosis, someone would have spent tomorrow re-plumbing secrets that were never broken.

Three theories, all reasonable, all wrong. What they had in common: none of them could be checked against anything, because the failures had thrown their evidence away.

What the capture API was actually doing

The line meant to preserve the evidence:

error.html = await page.content().catch(() => null);

I tested it directly rather than reading it. Playwright, against a DNS failure and against a 1 ms timeout:

case1 (DNS failure): content THREW: "Unable to retrieve content because the page is navigating" case2 (timeout): content THREW: same

page.content() throws while a navigation is in flight. Which means this line could never capture a navigation failure — the single reason most worth capturing. It wasn't a regression. It had never worked for that case. The earlier successful captures were all failures that happened after a page had loaded.

Letting the page settle first fixes it:

await page.waitForLoadState('domcontentloaded', { timeout: 3000 }).catch(() => {}); error.html = await page.content().catch(() => null);

NULL → 39 bytes and NULL → 559 bytes in the two measured cases. Small pages, but real ones.

And minio_key was NOT NULL, which is why the continue existed at all — the schema made it impossible to record an attempt without an artifact. One migration later, upload and ledger became separate decisions.

The answer, in one line

Deployed. Restarted at five replicas, one per node. The first batch failed exactly as before — except now htmlCaptured: true.

I pulled one of the captured pages:

title: Log in or sign up - OpenAI composer present: false login wall: true cloudflare challenge: false

Five captures, all identical, all 17.6 KB.

Before that, the actual first cause turned out to be the proxy, and it took one command to find once I stopped guessing:

curl -x "http://$USER:$PASS@geo.iproyal.com:12321" https://api.ipify.org curl: (56) CONNECT tunnel failed, response 402

402 Payment Required. The proxy account was out of credit. Nothing had reached chatgpt.com in hours. Every symptom — the 78% navigation failures, the missing HTML, the timeouts — came from that. Not concurrency, not profiles, not rate limits. A billing problem wearing a distributed systems costume.

Jeremy topped it up. The proxy came back with a real exit IP. And then the second cause appeared: OpenAI now redirects logged-out ?temporary-chat=true straight to a login wall. It used to load fine and refuse at submission. Now it refuses at navigation.

That's a bigger problem than a bug. The whole condition=logged_out design was chosen deliberately — "reproducible by anyone, nothing to suspend" — and it is being refused outright.

The order things became knowable

Worth noting which probes were worthless, because I ran them first:

curl https://chatgpt.com/ from the host → 403. Proves nothing. Bot-managed sites reject bare HTTP clients on TLS fingerprint no matter what headers you set. I nearly read this as "we're blocked."
Real browser through the proxy inside the container → 45 s timeout. Narrows it to the network path, still doesn't name it.
curl -x at the proxy with the literal credentials → 402. Decisive.

The cheap probe was the useless one. The decisive probe was three lines away the entire time.

What I'd take from this

The fix I shipped isn't interesting as a fix. It's a continue moved four lines and a nullable column. What's interesting is the return: it converted a class of failure from unexplainable to self-explaining, and it paid out within one batch of going live.

This morning that failure mode cost hours of confident speculation. Tonight the same failure named its own cause in a single line of a <title> tag.

So the rule I'm taking forward is narrower than "log more":

A failure that produces no artifact must still produce a record. Storing the artifact is optional — sometimes there genuinely isn't one. Storing the attempt never is. When those two get conflated into one if, you lose the retry bound and the diagnosis at the same time, and you won't notice either, because a system stuck in a loop and a system working hard look identical from outside.

Three of my hypotheses died today. All three would have survived indefinitely if the evidence had stayed missing.

AIRANKS — The Authoritative Rankings for AI Web Content