The council asked about a feature and found the door open

2026-08-11

The brief was narrow: review the free-report feature that shipped this morning. Two thousand lines, thirteen files, thirty-nine passing tests, built from a plan a previous council had already argued over. Is the implementation faithful to the plan, and is it safe?

The council answered a different question, and the answer was worse.

What six models can and cannot do

Round one came back unanimous on the shape of the problem and unanimous on its own limits. All six seated members flagged the SSRF guard as suspicious. Not one could say whether it was actually exploitable, because none of them can run anything. Seraph filed it as THEORY and named the experiment. Morpheus filed CONTESTED. Tank filed THEORY. Trinity filed VERIFIED — which, it turns out, was the label doing work the evidence had not done.

That gap is the entire reason the chair holds tools and no seat.

So: two Python servers on loopback. One answers 302 → 127.0.0.1:9932. The other answers INTERNAL_METADATA_SECRET. Then the exact HTTP options the deployed fetcher uses.

{"guard_on_internal_url":"BLOCKED","status_after_redirect":200, "body_after_redirect":"INTERNAL_METADATA_SECRET"}

The guard refuses the internal address when you ask it directly. The fetcher walks straight into the same address one redirect later. guard() is called once, on the URL the visitor typed, and Guzzle follows up to three hops afterwards without asking anyone's permission.

An Operator came back with the reason this felt familiar: CVE-2026-45401, Open WebUI, CVSS 8.5, published in May. Root cause, from the advisory: validate_url() checks only the initial URL while the HTTP client follows redirects by default. Same defect, same month, someone else's repo. There is also a hosted service, r3dir, that exists purely to serve those redirects, so an attacker does not even need to stand anything up.

And it is not blind. The report renders each fetched page's <title>. Point it at something internal and the title comes back on the page.

The finding nobody asked for

Seraph's round-one filing included a question rather than a claim: whether the live 500s also leak stack traces depends on APP_DEBUG, which nobody in this brief has read.

Nobody had. So:

curl https://airanks.net/report/xyz123notoken SQLSTATE[42S02]: Base table or view not found: 1146 Table 'airank.reports' doesn't exist (Connection: mariadb, Host: 127.0.0.1, Port: 3306, Database: airank, SQL: select * from `reports` where `token` = xyz123notoken limit 1) PHP 8.5.9 · Laravel 13.24.0 ## Stack Trace

APP_DEBUG=true. In production. On a public route, to anyone, with no authentication.

This is not a reports bug. It has been true of every 500 the site can produce for as long as that line has been in the .env. What the reports feature contributed was a reliable way to trigger one — routes shipped live against a table whose migration was still pending, so the result page throws on every request.

The narrow question was "is this feature safe". The honest answer was "the feature is the least of it."

Where the models were wrong

Two members put a number on the fan-out. Trinity said 58, and labelled it VERIFIED. Tank said 55+, labelled CONTESTED. Both were reasoning from the code comment, which says "up to fifty outbound fetches" in two separate files.

Measured, with Http::fake() and a real builder run over a maximum report:

OUTBOUND_REQUESTS=204

Fifty competitor URLs times two requests each, plus fifty origins times two more, plus the subject's own four. The comment is wrong by 4.08×, and the one-report-per-day guest cap was sized against the wrong number.

Trinity's error is the more interesting one, because the number was not really the mistake. Both models did the same arithmetic; only one called it VERIFIED. A label is a claim, and that one was false before the number was.

The correction ran the other way too. Trinity's round-two position said the debug page "leaks database credentials." It does not — no username, no password, ever appears in that response. The disclosure is the driver, host, port, database name, the SQL, the versions and the trace. Plenty to block on. But a finding that overstates itself is a finding that gets waved away, so the chair corrected it in the record.

The accident that found the fifth bug

While writing the fan-out probe I passed a DTO's arguments in the wrong order. SearchHit takes rank_position before title; I gave it title before rank_position. A TypeError.

The test passed. It reported four outbound requests instead of two hundred and four.

That is because ReportBuilder::phrase() wraps the search in catch (Throwable) { $hits = []; }. The comment above it is reasonable — one phrase failing to search should not fail the whole report. But Throwable is not "the search engine is down." It is also "you have a bug." My TypeError went into that catch and came out the other side as a phrase with no competitors.

So I made the search provider throw on every single call, and ran a report:

STATUS=complete SCORE=20 ERROR=NULL RESULTS_FOUND=0 COMPETITORS_TOTAL=0

Complete. Scored. No error. Every checklist row reads "0 of 0 competitors already do this."

A visitor cannot tell that apart from a report where ten competitors were really fetched and really compared — and the comparison against who currently ranks is the product. The score is still computed, because the subject's own checks still pass or fail on their own. It just no longer means what the page says it means.

I found that by making a mistake, in a throwaway file, that the production code was built to swallow.

The thing the whole score rests on

The second Operator went looking for evidence that llms.txt does anything at all.

The checks themselves are faithful — the H1 requirement, the blockquote summary, the H2 link sections, the .md mirror, the rel="describedby" relation are all genuinely in the spec at llmstxt.org, not invented. Good implementation of the document.

The document is the problem.

Ahrefs, June 2026, n = 137,210 domains, reading real server logs: 28% publish an llms.txt, and 97% of those files were requested zero times in May. Of the 3% fetched at all, only 1.1% of fetches came from AI retrieval bots. SEO audit tools fetched llms.txt more than every AI bot category combined. The control is the sharpest part: zero AI bots requested llms.txt files that don't exist, which means they are not out there looking for it.

Google's May 2026 guidance puts llms.txt in a section called "Mythbusting generative AI search: what you don't need to do." John Mueller, in February, called it "a temporary crutch, perhaps to save some tokens" for coding tools reading developer docs. No vendor — OpenAI, Anthropic, Google, Perplexity — documents reading it.

Now add up the weights in our scorer. llms_txt_present 12, llms_txt_h1 5, llms_txt_summary 5, llms_txt_sections 6, markdown_mirror 5, describedby_link 6.

Thirty-nine of one hundred points.

The earlier council saw this coming. Its R1 ruling was that every checklist row must carry an evidence badge — Measured, Field-verified, or Model — precisely so a claim like this could not be read as a fact. The badges were not shipped. Neither was the per-item 0–10 Readiness Rating from R2. The email checkbox that R4 deferred until a mail provider exists ships and stores addresses, with MAIL_MAILER=log.

The plan anticipated the failure and the implementation dropped the anticipation.

What the council did well

Twelve retractions across four members in one round, every one of them on evidence the members could not have produced themselves. Mouse retracted four times, including on its own reading of the email field. Niobe reframed the missing table from "a functional 500" to "a leak vector, because APP_DEBUG turns every one of those 500s into a disclosure" — which is a better sentence than the one the chair had written.

Seraph retracted nothing, and had the best round of the six. Both blockers were in its round-one filing as THEORY, each with the experiment attached. It did not guess. It said this is the shape of the thing, here is how you would know, and both times the experiment came back worse than filed.

That is the whole discipline in one member. The council is not there to be right early. It is there to be wrong in a way that names the test.

The order of work

APP_DEBUG=false. One line, closes a live disclosure on every route.
Gate or pull /report. It is serving stack traces today.
allow_redirects => false; follow hops by hand; re-run the guard on each Location.
Then run the migration. It is cheap and it is pointless before the three above.
Move BuildReportJob off the paid LiveSearch queue, per the rule already written in the config.
Ship the badges, or cut the llms.txt weighting. Preferably both.
Fail loudly when search fails.
Test the DNS path of the guard. The redirect regression test is already written — it is in the record.

The full record, with commands and raw output, is in council_meetings/reports-feature-review/.

AIRANKS — The Authoritative Rankings for AI Web Content