the files AI reads before it reads you

LLM Web Indexing Files

Researched from primary sources 13 Aug 2026 · every claim badged by evidence tier

Seven surfaces decide how AI systems discover, read, cite, and train on your site. This page breaks down each one: what it is, what it provably does, how to implement it, and — the part most guides skip — which benefits are VERIFIED and which are still THEORY. Every file below ships with a copy-paste starter. AIR's own domain reports audit all seven on any site.

All seven, ranked by what they actually do

Ordered by measured impact, not by chronology. If you only have an afternoon, do row 01 and stop.

The seven AI indexing surfaces compared by path, audience, effort and verdict
#	Surface	Path	Read by	Effort	Verdict
01	your HTML	(the page itself)	every AI crawler	hours–days	Do this before anything else
02	llms.txt	/llms.txt	coding agents, IDE tools	30 min	Worth the half hour
03	robots.txt	/robots.txt	every declared crawler	15 min	Decide it deliberately, once
04	sitemap.xml	/sitemap.xml	AI search crawlers	automate it	Generate it, keep lastmod honest
05	ai.txt	/.well-known/ai.txt	nobody yet	5 min	Cheap insurance, zero effect today
06	JSON-LD	in your <head>	search engines; AI unclear	half a day	No measured citation lift
07	ai-plugin.json	/.well-known/ai-plugin.json	a deprecated program	skip	Historical. Build an MCP server instead

How to read the badges

VERIFIED: a primary source confirms it: the spec, the vendor's own documentation, or a study with methodology.
COMMUNITY-REPORTED: practitioners observe it; no vendor confirmation.
THEORY: plausible, widely repeated, unproven. Not a reason to skip it — a reason to know what you're buying.

01 Render your content server-side #

The one that outranks the other six. Every file below is advice about your content. None of it matters if AI systems cannot read the content itself — and most of them cannot run your JavaScript.

Path: the page itself
Enforceable: n/a — it is your own HTML
Watch out for: <noscript> will not save you

Evidence

VERIFIED GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Meta's crawler fetch HTML but do not execute JavaScript (Vercel/MERJ crawler telemetry). Only Gemini's Google infrastructure and Applebot render JS. A client-rendered SPA reads as an empty shell to most of the AI ecosystem. Vercel & MERJ, 17 Dec 2024
VERIFIED<noscript> does not save you. AI crawlers ignore it.
VERIFIED We ate this one ourselves: our own summarizer judged airanks.net "essentially no substantive content", because this site is a client-rendered app. The fix — real server-rendered content in the initial HTML — shipped the same day, and the follow-up summary is on our own domain report. The audit tool must pass its own audit.

Do this

Server-render or statically generate anything you want cited. If that is a rewrite you cannot fund this quarter, ship a server-rendered floor: the headline, the summary, and the substance in the initial HTML.
Verify the way a crawler would — fetch without JavaScript and read what actually comes back.

verify like a crawler

# Read your own page the way an AI crawler does: no JavaScript, no mercy.
curl -sL -A "OAI-SearchBot" https://example.com/ | sed 's/<[^>]*>//g' | tr -s '[:space:]' ' '

# If that prints your nav and a footer and nothing else, the AI ecosystem sees
# nothing else either. Count the words you actually shipped:
curl -sL https://example.com/ | sed 's/<script[^>]*>.*<\/script>//g; s/<[^>]*>//g' | wc -w

02 llms.txt #

A curated markdown index of your best pages. Where robots.txt tells crawlers what they may not read, llms.txt tells AI readers what they should read.

Path: /llms.txt
Enforceable: n/a — it is an invitation
Watch out for: Google Search ignores it entirely

Proposed by Jeremy Howard (Answer.AI) in September 2024. The shape is fixed: an H1 title, a blockquote summary, then H2 sections of annotated links, with an ## Optional section for the skippable tail.

Evidence

VERIFIED The format and its specification are public and stable — llmstxt.org, proposed by Jeremy Howard (Answer.AI) in September 2024. Major AI vendors publish one for their own API documentation.
VERIFIED (negative) Google's John Mueller, on Bluesky, 17 June 2025: "FWIW no AI system currently uses llms.txt." He added that it is obvious from server logs — the consumer chatbots fetch your pages, but none of them fetch the file. Search Engine Roundtable, 18 Jun 2025
COMMUNITY-REPORTED That coding agents and IDE tools now fetch it at inference time, and that adoption sits somewhere around 6–9% of major sites. Widely repeated by practitioners; we have not found a primary source for either number, so we are not going to badge it as though we had.
THEORY That consumer chat assistants (ChatGPT, Claude, Perplexity) consult it when answering. No vendor has confirmed this. Publish it for the agent ecosystem that provably does.

Do this

Hand-write it. It is a curation, not an export — a generated dump of every URL defeats the point.
Keep it to about a screenful. Describe each link's payoff, not its title. Put your API and docs first.
Optionally add a larger llms-full.txt carrying the whole story (~1% adoption, cheap to add).

/llms.txt

# Example Corp

> Payments infrastructure for marketplaces. This file points AI readers at the
> pages worth reading; everything else is navigation.

## Docs
- [API reference](https://example.com/docs/api): every endpoint, with request
  and response examples. Start here if you are writing code.
- [Quickstart](https://example.com/docs/quickstart): first successful charge in
  about ten minutes.
- [Webhooks](https://example.com/docs/webhooks): event list, retry semantics,
  signature verification.

## Policy
- [Pricing](https://example.com/pricing): per-transaction rates by region.
- [Status](https://status.example.com): current and historical uptime.

## Optional
- [Changelog](https://example.com/changelog): dated release notes, 2019-present.
- [Engineering blog](https://example.com/blog): background reading, not reference.

Tooling: Mintlify auto-generates for docs sites; Yoast and AIOSEO ship WordPress plugins; validators exist at llmstxt.org. For most sites a text editor is the right tool.

03 robots.txt — the AI agent roster #

Training and search are separate decisions. The major vendors run different crawlers for different purposes, and blocking the wrong one costs you visibility without protecting anything.

Path: /robots.txt
Enforceable: no — a request, not a wall
Watch out for: Training and search are separate gates

The roster

AI crawler user-agents by vendor, purpose, and what blocking each one costs you
User-agent	Vendor	Purpose	Blocking it costs you	Honors robots?
GPTBot	OpenAI	Training	Nothing in search. This is the training opt-out.	VERIFIEDYes
OAI-SearchBot	OpenAI	Search index	Your listing in ChatGPT search.	VERIFIEDYes
ChatGPT-User	OpenAI	Live fetch	Live answers when a user asks about your page.	VERIFIEDYes
ClaudeBot	Anthropic	Training	Nothing in search. Training opt-out only.	VERIFIEDYes
Claude-SearchBot	Anthropic	Search index	Your listing in Claude's search results.	VERIFIEDYes
Claude-User	Anthropic	Live fetch	Live answers about your page.	VERIFIEDYes
Google-Extended	Google	Training gate	Gemini training only. Googlebot is untouched — this is a token, not a crawler.	VERIFIEDYes
Applebot-Extended	Apple	Training gate	Apple Intelligence training. Applebot search is untouched.	VERIFIEDYes
CCBot	Common Crawl	Open dataset	Inclusion in the open dataset many models train from.	VERIFIEDYes
Meta-ExternalAgent	Meta	Training / AI	Meta AI training.	VERIFIEDYes
PerplexityBot	Perplexity	Search + live	Your listing in Perplexity answers.	COMMUNITY-REPORTEDDisputed
Bytespider	ByteDance	Training	—	COMMUNITY-REPORTEDNo

Evidence

VERIFIED OpenAI's three agents have independent gates. Blocking GPTBot opts you out of training without removing you from ChatGPT search — and vice versa. Published IP ranges let you verify impostors. OpenAI's crawler documentation
VERIFIED Anthropic splits the same way — ClaudeBot, Claude-SearchBot, Claude-User — and Google-Extended is a robots token rather than a crawler, so blocking it never touches Googlebot. Anthropic · Google's crawler list
COMMUNITY-REPORTED ByteDance's Bytespider is repeatedly observed ignoring robots.txt despite vendor claims, and Cloudflare documented Perplexity fetching through undeclared headless browsers (2024). If opt-out matters to you, robots.txt alone is a request — WAF rules are the enforcement.

Do this

Decide training and search visibility separately. Write down which business you are in before you write a single Disallow.
Never block what you have not named. A bare Disallow: / under User-agent: * takes you out of every AI answer at once.
If opting out actually matters commercially, back it with WAF rules. The file is a request; the firewall is the enforcement.

/robots.txt

# Training and search are SEPARATE decisions. Decide them separately.

# --- OpenAI ---
User-agent: GPTBot                 # trains models
Disallow: /

User-agent: OAI-SearchBot          # the index behind ChatGPT search
Allow: /

User-agent: ChatGPT-User           # fetches your page when a user asks about it
Allow: /

# --- Anthropic ---
User-agent: ClaudeBot              # trains models
Disallow: /

User-agent: Claude-SearchBot       # search
Allow: /

User-agent: Claude-User            # live fetch
Allow: /

# --- Robots-only training gates (these are tokens, not crawlers) ---
User-agent: Google-Extended        # gates Gemini training; Googlebot is untouched
Disallow: /

User-agent: Applebot-Extended
Disallow: /

# --- Everyone else ---
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

This example opts out of training and stays in search. That is a choice, not a recommendation — invert the Disallow lines if your business runs the other way.

04 sitemap.xml #

The oldest discovery mechanism still works on the newest crawlers. Generate it mechanically and keep lastmod honest.

Path: /sitemap.xml
Enforceable: n/a
Watch out for: An honest lastmod, or none at all

Evidence

VERIFIED AI search crawlers discover through sitemaps referenced in robots.txt. The classic limits hold: 50,000 URLs / 50MB per file, index files above that.
VERIFIED — dated, and moving fast Cloudflare's crawl-economics data is the eye-opener. In July 2025 Anthropic's crawler fetched roughly 38,000 pages per referred visitor, OpenAI's about 1,100:1, against Googlebot's ~5:1. Cloudflare, Jul 2025
Do not quote that figure as current. The ratios have fallen by more than an order of magnitude since, as the vendors split search crawling from training. The durable finding is the direction — AI crawlers still read far more than they send back — not the number. Cloudflare publishes it live on Radar. How the ratio is measured
THEORY That an honest lastmod speeds AI citation refresh. Plausible, unmeasured. Keep it truthful anyway — lying to crawlers is how you train them to ignore you.

Do this

Generate from your router or CMS, never by hand.
Reference it from robots.txt with a Sitemap: line. The format is unchanged since 2005 — sitemaps.org/protocol.
Set lastmod to the date the content changed. Stamping today's date on every URL nightly is the fastest way to make the field worthless.

/sitemap.xml

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/docs/api</loc>
    <!-- The date this page's CONTENT changed. Not today's date. -->
    <lastmod>2026-07-14</lastmod>
  </url>
  <url>
    <loc>https://example.com/pricing</loc>
    <lastmod>2026-08-02</lastmod>
  </url>
</urlset>

05 ai.txt #

Nobody reads it yet. Publish it anyway — it costs five minutes and it is the legal breadcrumb you will wish you had.

Path: /.well-known/ai.txt
Enforceable: not technically — possibly legally
Watch out for: No major vendor reads it today

Spawning's machine-readable usage policy, now an IETF draft (draft-car-ai-txt-wellknown-00, June 2026). Where robots.txt is binary access control, ai.txt expresses nuance: crawl me, cite me, don't train on me — per-agent rules, license terms, attribution requirements.

Evidence

VERIFIED The format and its IETF draft status. The EU AI Act requires machine-readable opt-outs be honored, which is the regulatory tailwind behind it.
THEORYNo major vendor has announced reading it. Compliance today is approximately zero. This is cheap future-proofing, not an active control. We would rather say that plainly than sell you a file.

Do this

Serve the same short policy at /.well-known/ai.txt and /ai.txt — the draft location and the one older fetchers try.
State the license in one line a human can also read. Ours permits training and reproduction with attribution.

/.well-known/ai.txt

# https://example.com/.well-known/ai.txt
# Machine-readable usage policy. Also served at /ai.txt for older fetchers.

User-Agent: *
Allow: /
Disallow: /account/
Disallow: /checkout/

# Rights, in the order a lawyer would ask about them.
Training: n
Inference: y
Citation: required
License: https://example.com/terms#ai
Contact: legal@example.com

# In English, for the human who ends up reading this:
# Read us, quote us, cite us by name and link. Do not train on us.

06 JSON-LD structured data #

The most theory-encrusted item on this list. The best measured study found no citation lift at all. Ship it for search engines, not because someone told you it feeds the AI.

Path: in your <head>
Enforceable: n/a
Watch out for: JavaScript-injected markup is invisible

Evidence

VERIFIED (and sobering) The best measured study — Ahrefs, difference-in-differences across 1,885 pages that added JSON-LD against ~4,000 matched controls, Aug 2025 to Mar 2026 — found no citation lift: −4.6% in Google AI Overviews, +2.4% in AI Mode and +2.2% in ChatGPT, none of it statistically significant. No AI vendor documents JSON-LD as a citation signal. Linehan & Guan, Ahrefs, 11 May 2026
VERIFIED 53% of AI-cited pages carry JSON-LD — but that is correlation traveling with better technical quality generally. Google's July 2026 policy change also gates self-serving review and rating markup: publishing stars about yourself now risks eligibility rather than earning it.
VERIFIED (the trap) JSON-LD injected by JavaScript is invisible to AI crawlers. Tests across ChatGPT, Claude, Perplexity and Gemini fetches found none of them extracted JS-injected markup. If your schema comes from a tag manager, AI systems have never seen it.

Do this

Emit Organization and WebSite site-wide, and Article or FAQPage where they are actually true.
Render it from the server. If a tag manager injects it, delete it and start again.
Skip anything Google's policy now treats as self-serving — ratings about yourself most of all.
Validate what you ship: validator.schema.org. Tooling: spatie/schema-org (PHP), schema-dts (TypeScript).

Organization — server-rendered

<!-- Server-rendered. If a tag manager injects this, AI crawlers never see it. -->
<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Organization",
  "@id": "https://example.com/#org",
  "name": "Example Corp",
  "url": "https://example.com",
  "logo": "https://example.com/logo.png",
  "description": "Payments infrastructure for marketplaces.",
  "foundingDate": "2019-04-02",
  "sameAs": [
    "https://en.wikipedia.org/wiki/Example_Corp",
    "https://www.linkedin.com/company/example-corp"
  ]
}
</script>

07 ai-plugin.json #

Deprecated April 2024 — kept here so you can recognise it, not adopt it

The OpenAI plugin manifest at /.well-known/ai-plugin.json pointed agents at your OpenAPI spec. The plugin program it served was replaced by GPT Actions and, increasingly, MCP (Model Context Protocol). Its primary consumer no longer exists.

COMMUNITY-REPORTED Some 2026 agent runtimes still probe it as a fallback discovery path.

If you are building for agents in 2026, build an MCP server. That is where the ten minutes should go. The manifest below is a signpost to your OpenAPI spec, and the spec is the artifact that actually matters.

Show the historical manifest

/.well-known/ai-plugin.json

{
  "schema_version": "v1",
  "name_for_model": "example_payments",
  "name_for_human": "Example Payments",
  "description_for_model": "Look up payment and payout status for an Example Corp merchant by merchant ID or charge ID. Returns amounts in minor units and ISO-8601 timestamps.",
  "description_for_human": "Check payment and payout status.",
  "api": { "type": "openapi", "url": "https://example.com/openapi.yaml" },
  "auth": { "type": "none" },
  "contact_email": "support@example.com"
}

Note description_for_model: it is written for a model, not a human. That instinct is the part worth keeping — it is exactly how you write an MCP tool description today.

Audit any site against all seven #

The AIR domain report checks every surface on this page for any domain — presence, robots.txt AI-agent verdicts, server-rendered JSON-LD types, and live file contents — alongside the domain's AIR score and the AI's own summary of the site. No account needed.

Want to see one first? Read our own report — including the section where we failed our own server-rendering test. The build log documents how we fixed our files, before and after.

Before you close this: the seven checks

01your HTMLDo this before anything else
02llms.txtWorth the half hour
03robots.txtDecide it deliberately, once
04sitemap.xmlGenerate it, keep lastmod honest
05ai.txtCheap insurance, zero effect today
06JSON-LDNo measured citation lift
07ai-plugin.jsonHistorical. Build an MCP server instead

Run all seven automatically on any domain at airanks.net/d/<your-domain> — no account needed. Audited by ______________________ on ____ / ____ / ________

AIRANKS — The Authoritative Rankings for AI Web Content

All seven, ranked by what they actually do

01 Render your content server-side #

Evidence

Do this

02 llms.txt #

Evidence

Do this

03 robots.txt — the AI agent roster #

The roster

Evidence

Do this

04 sitemap.xml #

Evidence

Do this

05 ai.txt #

Evidence

Do this

06 JSON-LD structured data #

Evidence

Do this

07 ai-plugin.json #

Audit any site against all seven #