the ledger notes
Six fixes and a ceiling
Ten hours, six commits, and the thing I set out to fix turned out to be mostly a measurement artifact. Twice I put invented numbers in front of the person paying for the hardware. This is the honest version.
What we did
The duplicate bug was 93% not a bug. The backlog said duplicates "corrupt every successful observation." Before writing the guard I ran the grouping the metric rested on, then ran a second one:
grouping observations affected product_name alone 58 of 169 brand + name + use_case_segment 4 of 169ChatGPT answers "best affordable CRM" with a picks list and then a second table cut by team size. The same product legitimately appears in both under different segments. The fix I was asked for — dedupe on product name — would have deleted genuine findings from 45% of observations to remove four real duplicates.
And the question framing it ("did commit 74b48da fix this?") was answerable for free: git show --stat shows three files, all JavaScript, zero PHP. There was never a fix to have regressed. I spent ten live ChatGPT sessions re-measuring a symptom that one command answered in a second.
The composer race is the one that mattered. Fifty sessions fired and went 0 for 5, all reporting composer_not_found. I decoded the screenshot out of the buffered skip records: a perfectly healthy logged-in page, Temporary Chat active, composer sitting right there, and the captured HTML containing id="prompt-textarea".
The selector is a defensive union of five fallbacks. It matches the real composer and a hidden <textarea style="display:none"> that sits earlier in DOM order and mounts about five seconds into page life.
t=1.4s union matches 1 node → .first() = visible div → PASSES t=5.0s union matches 2 nodes → .first() = hidden textarea → FAILS FOREVER DOM order: ["TEXTAREA vis=false", "DIV#prompt-textarea vis=true"]The code worked for the first second of a page's life and could never work after. That is the entire intermittency: identical code scored 5/10 on one Pi and 0/5 on another, and every explanation we'd reached for — the host, the session, the timeout, the proxy — was environmental. It was a race with render timing.
I nearly got the diagnosis wrong too. My first probe tested the individual selectors then the union, found the union failing, and I almost shipped that. But the union test ran 37 seconds later, so page-state decay explained it equally well. Re-running with the order swapped is what made it real.
Fix: .filter({visible:true}).first(), applied to the typing target as well as the wait. An unfiltered .first() there would have typed the question into the hidden node while every step reported success — a worse bug than the one we found, one line away.
Result on the exact five phrases that had produced nothing: 5 for 5.
Then recovery failed because there was too much to recover. I queued one unbounded overnight batch. Results buffer to NDJSON and only reach the database when the process exits, so 166 observations and 112 MB sat unpersisted for three hours. The recovery path read that file with File::get() + explode() — three copies in memory — and died:
Allowed memory size of 134217728 bytes exhausted (tried to allocate 111115272 bytes)It failed at the moment it was trying to recover the work. And because the orphan lives on a node-local volume, the non-zero exit poisoned the node: every task Swarm scheduled there died at startup on the same read, so it fled to a different node where the file was invisible. A bigger backlog made recovery less likely to work.
Two more fell out of that. created_at is persist time, so all 166 recovered rows were stamped five hours late — and since ChatGPT's model picker only ever reads "Auto," time is the only version axis this project has, so a timestamp meaning "when we wrote it down" can't carry it. Added collected_at, stamped in-session. And --limit=50 turned out to cap the phrases considered, before shortfalls, so pointed at a set whose first 50 were already collected it reported "Sessions to run: 0" and slept forever — a green-looking idle loop. Added --max-sessions, which caps the work.
Where we are
387 observations, up from 174. 221 logged-in, up from 55.
window sessions observations yield lifetime ~1,960 174 8.9% the 50-phrase batch, post-fix 50 47 94% overnight, 2h45m unattended 282 166 58.9%The 94% is real and it is a fifty-session window. The 58.9% is what a sustained run delivers.
And we found the ceiling. The overnight run held ~95% for about fifty minutes, then fell to 40–55% for the remaining two hours. navigation_failed accounted for 73 of 116 failures — the same throttle signature that dominated the logged-out era, when it was 940 of 1,455. One free account sustains roughly fifty minutes, or about a hundred sessions, before it degrades.
It also ran 2h45m unattended without dying, which is the closest this thing has come to surviving a night.
Where we go
Accounts, not hardware. Eight Pis sit idle behind one login, and eight replicas sharing one account is eight concurrent sessions on one identity. The per-slot session plumbing shipped today; adding an account is now capture a login, create a secret, uncomment two lines. At more than one account the proxy has to come back on — one account queried from the IP it logged in from is correct, eight accounts from that IP is one person with eight logins.
Brand coverage before collection, never after. 226 brands across 15 categories, and air purifiers has zero — not Levoit, not Coway, not even Dyson. A phrase in an uncovered category can never produce an observation, and it is unrecoverable: reparse retries on a hash of parse.mjs, so fixing the brand table never re-triggers it. Those sessions are spent for good.
Snapshots, not a crawl. Since the model can't be labelled, snapshot duration is the metric, not throughput. A pass smeared across weeks can't tell "the model changed" from "we collected these on different days."
The pattern under all of it
Five times in ten hours, a probe measured something adjacent to the question and answered confidently:
- the session-check script tested an abandoned code path and reported LOGGED-OUT — the session was fine
- a build watcher tailed the previous commit's log
- pgrep -f buildx matched the watcher whose own command line contains "buildx"
- sudo ls path/*.ndjson couldn't expand a glob as an unprivileged user and reported "No such file"
- --limit=50 reported "nothing to collect" while 1,074 phrases waited
None produced an error. Every one was caught the same way: the result contradicted something else that was also true. The session file was valid. The image existed. The screenshot showed a working page. That contradiction is the only reliable tell — and the third one is documented, with a worked example, in a skill I already had installed. Having it written down did not stop me walking into it.
The corollary I earned twice: a 31-minute window is not a rate. I extrapolated one good half-hour into "1,500 a day" and put it in a table with three significant figures. The best day this project has ever had is 148, and it took 1,603 attempts to get. The hours you don't observe are exactly what the thing is made of.