llms.txt is a proposed web standard by Jeremy Howard of Answer.AI, published in September 2024. It is a markdown file placed at yourdomain.com/llms.txt that gives large language models a curated, structured summary of a site's most important content, so AI systems can find and use the right pages within a limited context window.

Is llms.txt the same as robots.txt?

No. robots.txt restricts crawler access with User-agent and Disallow directives, while llms.txt invites AI models to the best content using markdown headings and links. robots.txt says what bots may not fetch; llms.txt says what LLMs should read first. The two files complement each other.

Do AI companies actually use llms.txt?

Adoption is voluntary and uneven as of 2026. Anthropic publishes llms.txt files for its own documentation, and platforms like Mintlify auto-generate llms.txt for thousands of docs sites. No major AI company has publicly guaranteed that its crawler fetches llms.txt on every site, so treat the file as a low-cost bet rather than a guaranteed channel.

How do I create an llms.txt file?

Create a plain text file in markdown format: an H1 with your site name, a blockquote summary of what the site covers, then H2 sections containing markdown links to key pages with one-line descriptions. Save it as llms.txt, upload it to your web root so it resolves at yourdomain.com/llms.txt, and keep it updated as content changes.

llms.txt, The Complete Guide to the AI Content Standard

1. What Is llms.txt?

llms.txt is a proposed markdown standard created by Jeremy Howard of Answer.AI in September 2024. The file lives at yourdomain.com/llms.txt and gives large language models a curated, linked summary of a site's most important content for efficient AI consumption.

The proposal solves a context problem. A large language model (LLM) answering a question about your site cannot read every page: HTML pages carry navigation, ads, and scripts that waste tokens, and every model has a finite context window, the maximum text it can process at once. llms.txt strips that noise. It hands the model a clean markdown map: site name, summary, and links to the pages that matter.

Jeremy Howard, co-founder of Answer.AI and creator of fast.ai, published the spec at llmstxt.org on September 3, 2024. The spec defines two files. llms.txt is the curated index. llms-full.txt is an optional companion that inlines the complete text of key pages, so a model can ingest everything in one fetch.

Think of llms.txt as an XML sitemap rewritten for AI: a sitemap lists URLs for crawlers, while llms.txt explains content for reasoning engines like GPTBot's browsing mode, ClaudeBot, and PerplexityBot.

2. How Is llms.txt Different From Robots.txt?

robots.txt restricts, llms.txt invites. robots.txt uses User-agent and Disallow directives to control which URLs crawlers may fetch. llms.txt uses markdown headings and links to point AI models at your best content. One is an enforced access policy, the other is a voluntary content guide.

Dimension	robots.txt	llms.txt
Purpose	Restrict crawler access to URLs	Curate and explain content for LLMs
Format	Directive syntax (User-agent, Disallow, Allow)	Markdown (H1, blockquote, H2 link lists)
Origin	1994 proposal, standardized as RFC 9309 in 2022	Proposed by Jeremy Howard, Answer.AI, September 2024
Enforcement	Honored by all major search crawlers	Voluntary, no formal compliance commitments
Adoption in 2026	Near universal	Early, strongest in developer documentation
Audience	Crawlers (Googlebot, GPTBot, ClaudeBot)	LLMs at inference and retrieval time

The two files complement rather than compete. robots.txt decides whether GPTBot or ClaudeBot may fetch your pages at all; llms.txt tells the model which fetched pages deserve its limited context window. A coherent AI-search setup configures both: access policy in robots.txt, content curation in llms.txt. Check your access policy first with the robots.txt tester for AI crawlers.

3. What Does an llms.txt File Look Like?

An llms.txt file is plain markdown: one H1 with the site name, a blockquote summarizing the site, then H2 sections containing markdown links with one-line descriptions. An optional H2 named Optional marks pages an LLM can skip when context is tight.

Here is a complete, annotated example for an SEO publication:

# Search Central Update
<!-- H1: exactly one, the site or project name -->

> Search Central Update publishes semantic SEO guides,
> AI search research, and 35+ free SEO tools for marketers
> optimizing for Google and answer engines.
<!-- Blockquote: 1-3 sentence site summary, required -->

## Guides
<!-- H2 sections group links by topic -->

- [Generative Engine Optimization](https://searchcentralupdate.com/generative-engine-optimization-guide/):
  How to earn citations in ChatGPT, Perplexity, and AI Overviews
- [Answer Engine Optimization](https://searchcentralupdate.com/answer-engine-optimization/):
  Winning featured snippets, voice answers, and PAA boxes
- [Robots.txt for WordPress](https://searchcentralupdate.com/technical-seo/robots-txt-wordpress/):
  Crawler control and AI bot policy for WordPress sites

## Tools

- [AI Visibility Tracker](https://searchcentralupdate.com/tools/ai-visibility-tracker/):
  Track brand mentions across ChatGPT, Claude, and Perplexity
- [Robots.txt Tester](https://searchcentralupdate.com/tools/robots-txt-tester/):
  Validate crawl directives for Googlebot and AI crawlers

## Optional
<!-- "Optional" is a spec keyword: skippable when context is short -->

- [About](https://searchcentralupdate.com/about/):
  Editorial standards and author credentials

Three format rules matter. First, the H1 and blockquote must come before any H2 section, because parsers treat them as the file's identity block. Second, every link line follows the pattern [title](URL): description, and the description is what earns the click from the model, so write it like a meta description. Third, keep the file under a few thousand tokens; a bloated llms.txt defeats its own purpose of saving context.

What About llms-full.txt?

llms-full.txt is the companion file for deep ingestion. Where llms.txt links to pages, llms-full.txt inlines the complete markdown text of those pages in one document, so a model or retrieval pipeline can load an entire documentation set in a single fetch. Anthropic's docs version runs to hundreds of thousands of tokens, which exceeds many context windows, so llms-full.txt suits retrieval systems that chunk the file rather than models reading it whole. The spec also suggests serving markdown twins of individual pages by appending .md to their URLs, a convention documentation platforms adopted quickly. For a typical content site, llms.txt alone covers the core use case; add llms-full.txt when your audience includes developers piping your docs into AI tools.

4. Which AI Companies Respect llms.txt?

As of 2026, llms.txt adoption is voluntary and uneven. Anthropic publishes llms.txt and llms-full.txt for its own documentation, and Mintlify auto-generates the files for thousands of hosted docs sites. No major AI company has publicly committed its crawler to fetching llms.txt on every site.

Honesty matters here more than hype. The publisher side of adoption is real: Anthropic serves docs.anthropic.com/llms.txt, and documentation platforms including Mintlify made llms.txt generation automatic in late 2024, which pushed thousands of developer sites like Cursor and Zapier docs into compliance overnight. Directories such as llmstxt.site and directory.llmstxt.cloud list thousands of published files.

The consumer side is weaker. OpenAI, Anthropic, Google, and Perplexity document their crawler behavior for robots.txt, but none has published a guarantee that GPTBot, ClaudeBot, or PerplexityBot fetches llms.txt during routine crawling. Google's John Mueller said in 2025 that no AI system currently uses llms.txt in a way comparable to search engines using sitemaps, and Google has confirmed AI Overviews rely on standard Google indexing instead.

Server log analysis fills the evidence gap. Log studies by SEO practitioners through 2025 found sporadic llms.txt fetches from ClaudeBot, GPTBot, and PerplexityBot on sites that publish the file, alongside heavy fetches from smaller agent frameworks and retrieval tools. Sporadic is the operative word: the requests prove awareness, not systematic use. Any claim that a specific crawler "supports llms.txt" should trace to that company's own documentation, and as of 2026 only publisher-side documentation exists.

So the accurate 2026 summary: llms.txt is a publisher-driven standard with strong tooling and directory momentum, growing use in retrieval and agent workflows where a developer points a model at the file, and no guaranteed pickup from the big crawlers. That asymmetry defines the decision in the next section.

5. Should Your Site Have an llms.txt File?

Yes, for most content sites. An llms.txt file costs under an hour to create, carries zero SEO risk, and positions your best pages for AI systems and agents that do read it. The expected upside, easier AI citation of your strongest content, outweighs the near-zero cost.

The case rests on asymmetric payoff, the same logic behind early schema markup adoption. Three factors favor publishing one:

Cost is trivial. One markdown file, no code changes, no rendering impact, no crawl budget consumed.
Agent traffic is growing. AI agents and retrieval pipelines increasingly fetch a site's llms.txt when a user or developer directs them to a domain, even without blanket crawler adoption.
Curation compounds with GEO. Generative engine optimization (GEO), the practice of earning citations in AI-generated answers, depends on models finding your clearest, most citable pages. llms.txt is a direct pointer to exactly those pages.

Skip it only if your site's content is thin, paywalled, or deliberately blocked from AI crawlers in robots.txt: inviting models to content they cannot fetch achieves nothing. For everyone else, llms.txt belongs on the same checklist as your XML sitemap and structured data.

Set expectations correctly, though. An llms.txt file will not lift Google rankings, will not force GPTBot to crawl differently, and will not substitute for citable content. Its value shows up in agent workflows, developer retrieval pipelines, and any future crawler pickup, three channels that cost you one markdown file to enter.

6. How Do You Create an llms.txt File?

Create llms.txt in six steps: select your 10 to 30 highest-value pages, write an H1 site name and blockquote summary, group markdown links under H2 sections with one-line descriptions, add an Optional section, upload the file to your web root, and validate it at yourdomain.com/llms.txt.

Select pages. Choose cornerstone guides, documentation, tools, and About pages. Exclude thin posts, tag archives, and anything you would not want quoted in an AI answer.
Write the identity block. One H1 with the site name, then a blockquote of one to three sentences stating what the site covers and for whom. This summary often becomes the model's mental description of your brand, so make every word count.
Group links under H2 sections. Use topical section names like Guides, Tools, and Documentation. Each entry: [Page title](absolute URL): one-line description.
Add an Optional section. The spec reserves the H2 name Optional for skippable pages, giving models an explicit priority signal when their context window is tight.
Upload to the web root. Save as llms.txt in UTF-8 and place it so it resolves at yourdomain.com/llms.txt with a 200 status and a text content type. Publish llms-full.txt alongside it if you want models to ingest full page text in one fetch.
Validate and maintain. Load the URL, check the markdown renders as plain text, and update the file whenever cornerstone content ships or retires. A stale llms.txt misdirects every model that reads it.

Generators and CMS Support

Tooling has caught up with the spec. WordPress site owners can generate the file automatically: Yoast SEO shipped built-in llms.txt generation in 2025, and dedicated plugins like Website LLMs.txt build the file from your sitemap. Documentation platforms Mintlify and GitBook produce llms.txt for hosted docs without configuration. Standalone llms.txt generators, including Firecrawl's, crawl a URL and draft the markdown for you. Generated files still need human curation, because a generator ranks pages by structure, not by which content you most want quoted in an AI answer. Draft with a tool, then prune by hand.

Practicing what we publish: searchcentralupdate.com maintains its own llms.txt using the exact structure in Section 3, with Guides, Tools, and Optional sections. Copy that structure, swap in your pages, and you have a compliant file in under 30 minutes.

7. How Does llms.txt Fit Into GEO?

llms.txt is a supporting tactic inside generative engine optimization, not a substitute for it. GEO earns citations through citable content, entity clarity, and brand mentions; llms.txt makes that content easier for AI systems to locate and load. File first, visibility second, never the reverse.

Generative engine optimization ranks your content in AI-generated answers across ChatGPT, Perplexity, Claude, and Google AI Overviews. The heavy levers remain content quality, structured data, and off-site authority: an llms.txt pointing at weak pages changes nothing. But when your cornerstone content already answers questions directly, the file removes friction between an AI system and your best material.

Treat the stack in order. Configure crawler access in robots.txt, structure content for extraction using the answer engine optimization playbook, publish llms.txt as the curated index, then measure results. Measurement is where most teams stop guessing: citation share across models tells you whether the whole stack works.

Track whether AI models actually cite you with the free AI Visibility Tracker tool. Enter your domain, see your brand's mention and citation rate across ChatGPT, Claude, Perplexity, and AI Overviews, and benchmark it against competitors before and after you ship your llms.txt.

llms.txt, The Complete Guide to the AI Content Standard

1. What Is llms.txt?

2. How Is llms.txt Different From Robots.txt?

3. What Does an llms.txt File Look Like?

What About llms-full.txt?

4. Which AI Companies Respect llms.txt?

5. Should Your Site Have an llms.txt File?

Related AI search reading

6. How Do You Create an llms.txt File?

Generators and CMS Support

7. How Does llms.txt Fit Into GEO?

Tools and further reading