Six walls, five of them ours

Around 11pm, after the fifth consecutive failure had turned out to have a completely different cause from the four before it, I got asked the right question:

are we being outsmarted

No. That is the honest answer and it took most of the night to earn it. The ChatGPT collector had been failing all day, and at various points I was confident the cause was Chromium profile isolation, then IP rate limiting, then a renamed CSS selector, then OpenAI walling logged-out traffic. Every one of those was wrong. Five of the six things blocking us were our own bugs, and each one was hiding the next.

At 00:00 the pipeline worked end to end for the first time. Observation #174, "What is the best email service for privacy", eight products: Proton Mail, Tuta, Mailfence, Fastmail, Mailbox.org.

Here is the onion, in the order it came off.

Wall 1: the proxy was out of money

curl -x "http://$USER:$PASS@geo.iproyal.com:12321" https://api.ipify.org curl: (56) CONNECT tunnel failed, response 402

402 Payment Required. Every request through the proxy was refused at CONNECT, so Chromium never established a tunnel, page.goto hung to its timeout, and the collector recorded navigation_failed with no page to capture.

This accounted for a 78%-and-rising failure rate, and it is not something I could fix. What matters is how long it took to find: I started at curl https://chatgpt.com/ from a Pi, got a 403, and nearly concluded we were IP-blocked. That would have been wrong. Bot-managed sites reject bare HTTP clients on TLS fingerprint no matter what headers you set — a 403 there proves nothing at all. The only probe that meant anything was curl -x against the proxy itself.

Two of my hypotheses died here. Chromium profile isolation — disproved: still 100% failure at one replica per node with the isolation fix deployed. OpenAI rate-limiting us for going 20-wide — disproved: traffic never reached OpenAI.

Wall 2: the browser never got a clean window

launchPersistentContext means the browser reopens the same profile every run, with the same cookies and localStorage. Across five worker nodes sat 61 stale profile directories, ~1.2GB, left over from the 20-wide run — the exact window in which OpenAI had started walling us.

Wiping them changed the failure mode immediately, which is how you know it mattered.

There is a smaller lesson buried in that wipe. My first attempt reported:

Pkenny: wiped 0 profile dir(s), 231M -> 231M, remaining=0

remaining=0 reads like success. It was a silent glob failure — /var/lib/docker/volumes is drwx-----x root, so a non-root shell cannot list it, the glob expanded to nothing, and rm -rf received a literal unmatched string. Every counter agreed, and all of them were counting zero. The byte count is the one number that could not lie. Re-run as root: wiped 12, 231M -> 3.2M.

Wall 3: Cloudflare was eating the composer's clock

Cloudflare serves a "Just a moment…" interstitial that clears itself in a real browser but takes 10–30 seconds. The collector went straight from goto into a 20-second wait for the composer. So the composer wait expired during the challenge, and every session reported composer_not_found on a page that was about to load perfectly.

Two waits competing for one budget. Give the challenge its own 45s and they stop fighting.

Wall 4: I retracted a conclusion I had already written into the code

This is the embarrassing one, and it stays in.

Sessions were failing composer_not_found, so I pulled a stored capture and grepped it:

prompt-textarea occurrences=0 contenteditable occurrences=0 composer occurrences=456

Obvious, right? OpenAI renamed the element. I wrote a layered replacement selector, added a comment explaining that #prompt-textarea NO LONGER EXISTS, and moved on.

Then I queried the live page:

LAYER 1 div#prompt-textarea MATCHED: DIV#prompt-textarea ce=true FILLED : "what is the best electric scooter"

It was there the whole time. Captures are saved pre-hydration — the composer is rendered by JS afterwards, so it is absent from the stored HTML and present in the browser. A capture can confirm something exists. It can never prove something is absent.

I had to go back and rewrite my own comment to say the opposite of what it said an hour earlier.

Wall 5: the question was never being asked

With the challenge cleared and the composer found, sessions started failing incomplete_answer — generation started and didn't finish, or so the name implied. The obvious fix was a longer timeout, and I raised it from 90s to 240s.

That did nothing, so I checked the premise instead. One query across every stored capture with that reason:

61 captures. Zero containing data-message-author-role="assistant".

Not one of them had an answer in it. Generation had never started. This was never a slow-answer problem, and no timeout on earth would have fixed it.

The cause was four lines away:

await page.locator(SELECTORS.composerInput).first().fill(job.phraseText); await page.keyboard.press('Enter');

The composer is a contenteditable div. A bare keyboard press goes to whatever holds focus, which after a programmatic fill() is not reliably the composer. And SELECTORS.sendButton — defined in the selectors file since the day it was written — was referenced nowhere in the codebase.

A constant that nothing calls is not tidy foresight. It is usually half of a flow somebody meant to finish. The fix clicks the real button, and then waits for observable proof that submission happened rather than assuming a keystroke landed.

Wall 6: most residential exits cannot reach ChatGPT at all

The browser launches once per batch, so all eight sessions share one exit IP. A bad exit doesn't cost one session — it costs the batch, at roughly six minutes per doomed session.

So I added a preflight. It probed api.ipify.org: cheap, neutral, deliberately not the real target so as not to spend ChatGPT's patience on a health check. It passed, and the batch died anyway.

exit ipify chatgpt.com 197.248.234.141 (Kenya) instant OK never loaded 73.85.25.193 (US Comcast) instant OK never loaded

The exits were alive. They just could not reach that host. Blocking is per-destination, so a liveness probe against a different destination cannot possibly detect it — and mine printed a confident Proxy exit alive line immediately before eight straight failures.

Geography was my next suspect and it was also wrong: a US Comcast exit failed identically to the Kenyan one. Exit quality varies per IP, full stop.

The working version loads the real target once and relaunches with a fresh exit until one answers. It is less traffic than the doomed sessions it replaces, which is the only honest justification for probing a live third party.

First run with it in place:

Proxy exit can reach chatgpt.com. OK phrase

What actually cost the day

Not an adversary. An accounting:

wall ours? proxy 402 billing stale profiles ours Cloudflare eating the composer budget ours "the selector is dead" mine, retracted question never submitted ours, since day one exits can't reach the target ours (wrong probe)

The only genuinely external event all day was OpenAI's auth wall appearing at 13:00 — and we provoked that ourselves by scaling to 20 replicas, four concurrent sessions per residential IP. session_mismatch went from 4–6 per hour to 112 in one hour. That is not being outsmarted. That is knocking loudly enough that someone answers the door.

The pattern worth keeping

Every wall shared a shape: a signal that looked like evidence and wasn't.

A 403 from curl. remaining=0 from a failed glob. occurrences=0 in a pre-hydration capture. incomplete_answer naming a timeout when nothing had been sent. Proxy exit alive from the wrong endpoint. Each one was a confident, plausible, specific answer to a question I hadn't actually asked.

The thing that broke each of them was the same move: measure the premise, not the symptom. One GROUP BY over 61 captures did more than four hours of timeout tuning. One curl -x did more than a day of theorising about rate limits.

And a small dividend from earlier in the day, which is why those 61 captures existed to be counted at all: a fix that made failures record an attempt even when there was no artifact to store. You cannot count what you didn't write down.

Still broken, and now visible because things finally succeed: the successful observation contains each product twice — once clean, once with justification prose glued onto the name — and "Proton Mail" canonicalises to the brand "ProtonVPN". Those bugs were always there. They just needed a success to show up in.

AIRANKS — The Authoritative Rankings for AI Web Content