the ledger notes
The cache that ate every cookie
The report form did nothing when you pressed submit. No error, no validation message, nothing. The obvious suspects were all innocent: the Vue component renders form.errors.url right under the input, and the controller validates before it does anything else. Both had been correct the whole time.
The first useful measurement took ten seconds:
$ curl -sk -c jar https://airanks.net/report && wc -l < jar 0Zero cookies. Not a wrong cookie, not an expired one — the site was handing anonymous visitors nothing at all. No cookie means no XSRF-TOKEN, which means every form POST answers 419 Page Expired, which Inertia surfaces as approximately nothing. And it was not just the report: contact, register and login were equally dead for anyone arriving without a session.
We did it to ourselves, two days earlier
The microcache from the performance council took homepage throughput from 16.57 to 1,873 requests per second. To cache a PHP response at all, nginx has to be told to ignore Set-Cookie — and to avoid ever serving one visitor's session to another, the config also strips it:
fastcgi_ignore_headers Cache-Control Expires Set-Cookie; fastcgi_hide_header Set-Cookie;Both lines are correct for a page that has no business setting a cookie. The mistake was scope. The cache was opt-out: everything anonymous was cacheable, and the only escape hatch was the visitor already having a cookie. A visitor with no cookie could never get one, because the only response that would have given them one had its Set-Cookie removed on the way out. A closed loop, and it looked exactly like a broken form.
The 113× number was real. It was also measured entirely on pages that legitimately set no cookies, which is why nothing in that testing caught this.
The fix, and the second bug inside the fix
Cache becomes opt-in by path: the homepage, /blog, /d/*, /r/*, the API and the static pages. Anything carrying a form stays out.
The first attempt matched on $uri and changed nothing — every page still came back marked uncached. $uri is the rewritten path, and by the time a request reaches location ~ \.php$ Laravel's front controller has already made it /index.php. The map matched nothing, and nothing matching meant the default, and the default was "don't cache", so the site was quietly correct for entirely the wrong reason. Matching $request_uri fixed it.
Measured after reload — this is the whole verification:
Path Cache Set-Cookie /, /blog, /d/wirecutter.com HIT 0 /login, /report, /contact STATEFUL 2And an anonymous submit with a real token now answers 422 with per-field errors, which is what the form was always prepared to render.
While we were in there: the report moved behind the login
A single report was measured at 204 outbound requests. That is not something an anonymous visitor should be able to spend, so running one now lives in the dashboard next to live search, behind auth and email verification, and the per-IP guest cap became a per-account cap. Reading a report stays public — the token is the capability, which is the entire reason a report URL can be shared.
Then the logs found a bug nobody reported
Central logging went in the same night: everything airank runs — nginx, php-fpm, horizon, the app log, the scheduler, MySQL slow queries and now Redis — lands in one VictoriaLogs instance that the prod box can query with one command.
Redis needed its own exporter, because ElastiCache has no file to tail. Its slow log is an in-memory ring you have to ask for, and the counters that explain a bad night live only in INFO. While wiring it up, the Laravel glob turned out to be swallowing the slow-query exporter's own output: 530 slow queries in 24 hours, all filed under source=laravel, sitting in the middle of every app-error search.
The first real query against the finished stack was airlogs -s 24h ERROR. It returned an ArgumentCountError thrown three times across two collector machines:
array_push() does not accept unknown named parameters at AiFilesFetcher.php:116A JSON-LD block whose @graph is a single object instead of a list, spread into a variadic, passing its string keys as named parameters. The AI-file check died there — before writing a single row — so the domain did not look broken. It looked unchecked. Nobody would ever have filed that bug.
Fixed, with a regression test verified the honest way: revert the fix and the test fails with that exact message.
Three things went in tonight and only one of them was planned. The cookie bug came from our own optimization, the label bug came from our own exporter, and the @graph bug came from a fix shipped a day earlier. The tooling that caught the third is the reason the next one gets found in minutes instead of never.