llms.txt: what it is and whether it matters

llms.txt is a proposed Markdown file, placed at /llms.txt, that gives AI models and agents a short guide to a site and links to its key pages (opens in a new tab). Google says Google Search doesn't use such files (opens in a new tab), and in one vendor's logs from 137,210 domains, 97% of published files got no requests in May 2026 (opens in a new tab). AEO HQ recommends a file only if AI coding tools read your documentation or a partner asks for one. Do not expect it to raise rankings or citations.

The file is one of the smaller topics in AI search optimization. This guide explains the proposal, who reads the files, what platforms say, and when a file is worth keeping. It separates documented facts, measured results, and AEO HQ's recommendations. Its sources were checked on September 27, 2026.

Scope and definitions

• llms.txt is the file itself: a plain-text Markdown file at the root of a site, or under a path such as /docs/llms.txt.
• Markdown is a plain-text format that marks headings, lists, and links with symbols such as #. The proposal uses it because "we expect many of these files to be read by language models and agents" (opens in a new tab).
• An AI agent is software that carries out a task for a person. Google describes agents as "autonomous systems that can perform tasks on behalf of people, such as booking a reservation or comparing product specifications" (opens in a new tab). A coding assistant that reads a library's documentation is one kind.
• robots.txt tells crawlers which pages they may fetch. llms.txt is not a replacement: it "controls nothing and blocks nothing" (opens in a new tab).
• A sitemap lists a site's pages for search engines. The proposal describes the difference this way: sitemaps list all pages for search engines, while llms.txt "offers a curated overview for LLMs" (opens in a new tab).

This guide covers the llms.txt file and the Markdown page copies that the proposal describes. It does not cover the robots.txt rules that tell AI crawlers which pages they may fetch; those are covered in how AI assistants find sources.

How llms.txt works

What the proposal says

The proposal was first written in 2024 (opens in a new tab), by Jeremy Howard, co-founder of Answer.AI and fast.ai (opens in a new tab). Its current version sets out these parts (proposal document (opens in a new tab)):

• Location. The file sits at the site root or at any path, and covers the pages under that path.
• Format. An H1 with the site's name, which is the only required part; a blockquote with a short summary; optional paragraphs or lists without headings; and sections under H2 headings that list links, each with an optional note. A section headed "Optional" holds links an agent can skip.
• Markdown copies of pages. Pages that agents may need can offer a clean Markdown version at the same URL with .md added.
• Discovery. Pages can point to the file with a rel="describedby" link, and to their Markdown copy with rel="alternate" type="text/markdown", in HTML or in an HTTP Link header.

The proposal is clear about its purpose. llms.txt information "is instead used on demand, when an agent needs information about a topic while assisting a user" (opens in a new tab), and its author expected it to be useful "mainly" for "inference rather than training" (opens in a new tab). By its own account, llms.txt files are used most heavily for software documentation, where coding agents follow them to find API references and tutorials (opens in a new tab).

A short file in the proposal's format looks like this. The company and links are placeholders:

# Example Co

> Example Co sells fixed-price bookkeeping to software companies. [One or two sentences with the facts buyers ask about most.]

## Services
- [Bookkeeping](https://www.example.com/bookkeeping): scope, deliverables, and monthly price
- [Tax filing](https://www.example.com/tax): what is included and turnaround

## Company
- [About](https://www.example.com/about): founders, location, and contact details

## Optional
- [Articles](https://www.example.com/articles): guides for software companies

Who reads llms.txt files

A log study by Ahrefs, a vendor that sells the analytics used to collect the data, covered 137,210 domains that received traffic in May 2026 (opens in a new tab):

• Adoption. 28% of the domains, about 38,000, published a valid file (opens in a new tab). Ahrefs treats this as an upper bound, because its customers "skew more technical and SEO-aware than the web at large" (opens in a new tab).
• Reads. 97% of those files received no requests at all in May 2026 (opens in a new tab). The other 3%, about 1,100 domains, received about 22,000 requests in total (opens in a new tab).
• Who asked. 96% of those requests came from bots and 4% from people (opens in a new tab).

The bots that fetched the files were mostly not AI search systems (vendor study (opens in a new tab)):

Four details stand out:

• OAI-SearchBot, PerplexityBot, and Claude's search crawler together made "only a couple of hundred fetches across thousands of sites" (opens in a new tab). These are the search crawlers of ChatGPT, Perplexity, and Claude; see OpenAI's, Anthropic's, and Perplexity's crawlers in the glossary.
• Anthropic's coding agent, Claude-Code, fetched the files more often than every AI search crawler and assistant (opens in a new tab).
• Slackbot, which builds link previews in a chat app, fetched llms.txt files more often than PerplexityBot did (opens in a new tab).
• No AI bot requested an llms.txt file that did not exist; 98% of requests for missing files came from people (opens in a new tab).

Ahrefs adds a caution that applies to every figure above: a fetch is not proof that anything read or acted on the file, so each figure is "a ceiling on actual llms.txt consumption" (opens in a new tab).

What platforms say

None of the crawler pages from OpenAI (opens in a new tab), Anthropic (opens in a new tab), or Perplexity (opens in a new tab) asks site owners to publish a file. OpenAI and Perplexity publish their own, for their developer documentation.

Google's position has two parts that can look like a contradiction. Google Search ignores the file, while Chrome's Lighthouse tool checks for it and says that without it, "agents may spend more time crawling the site to understand its high-level structure and primary content" (opens in a new tab). Asked on Bluesky why Google's developer site publishes such files, Google's John Mueller replied that "it's not done for search," called the files "more of a temporary crutch, perhaps to save some tokens" for AI coding tools, and said that for non-developer sites "I don't think this makes much sense" (opens in a new tab) (a Google employee's social media post, reported by a trade publication).

What studies of effects show

• Citations. Across nearly 300,000 domains (vendor study, November 2025), 10.13% had a file, and removing it from a model of how often each domain was cited by AI made the model more accurate (opens in a new tab), meaning the file added no predictive value. The article does not say which AI systems or prompts were used.
• A laboratory test. C-SEO Bench, a peer-reviewed benchmark, tested "LLM Guidance," a method "inspired by the LLMs.txt standard" that adds a Markdown summary to the start of a document (opens in a new tab). It raised rankings significantly only for product recommendations in the retail and video game data, on one model, GPT-4o-mini; no method worked for question answering, and none worked on Claude 3.5 Haiku (opens in a new tab). Across the whole benchmark, only 3 of 54 tested cases showed significant gains (opens in a new tab). The test put the summary inside each document. It did not test a separate file at /llms.txt.

Risks

• Prompt injection. Ahrefs found that the largest research crawler requesting llms.txt files identified itself as "prompt-injection-survey/1.0" (opens in a new tab), and warns that a stale or compromised file "misleads every agent that reads it" (opens in a new tab).
• Manipulation. Writing instructions to AI systems into the file falls under the same rules as hidden instructions anywhere. Google's spam policies cover attempts "to manipulate generative AI responses in Google Search" (opens in a new tab), and Bing says content designed to manipulate language models "may result in reduced visibility or removal from search experiences" (opens in a new tab).
• Drift. A file that states old prices or services contradicts the site. Inconsistent facts are covered in answer engine optimization antipatterns.

Steps

These are AEO HQ's recommendations. The facts behind them are linked.

1. Fix crawling and indexing first. AI search crawlers rarely fetch llms.txt (opens in a new tab), while Google lists allowing crawling "in robots.txt, and by any CDN or hosting infrastructure" among the basics for its AI features (opens in a new tab). Check the robots.txt rules, firewall settings, and server-rendered text that decide whether assistants can reach your pages; the planned technical SEO checklist for AI search and SEO antipatterns cover them.
2. Decide whether you need a file. Publish one if developers or AI coding tools use your documentation, or if a partner asks for it, as Stripe's Directory does for agent-payable services (opens in a new tab). How agents find and pay for services is covered in how AI agents find and buy services. For other sites, the file is optional: it costs little, and Google says it neither helps nor harms visibility in Google Search (opens in a new tab).
3. Write it to the proposal's format, and keep it short. Use an H1 with the company name, a one-paragraph summary, and links with one-line notes to the pages that answer buyers' main questions. Every fact must match the linked page word for word.
4. Point agents to it. Add a rel="describedby" link or an HTTP Link header, as the proposal describes (opens in a new tab). Ahrefs found that agents fetch the file "when directed, not speculatively" (opens in a new tab).
5. Treat it like code. Ahrefs recommends that sites version-control the file, restrict who can edit it, alert on unauthorized changes, and keep it to plain links and descriptions, with "nothing instruction-shaped" (opens in a new tab). Review any file a platform generates for you.
6. Check your logs after 30 days. Count requests to /llms.txt by user agent. Ahrefs suggests checking your own logs before investing further (opens in a new tab).
7. Report it as housekeeping. No study links the file to AI citations, so a plan or report should not present it as an AI visibility result.

What the evidence shows and does not show

Antipatterns

An antipattern is a practice that looks helpful but fails or backfires. The same entry, with a detection test, is in SEO antipatterns.

1. Counting llms.txt as an AI visibility result. 97% of files received no requests (opens in a new tab), and Google Search ignores them (opens in a new tab). Instead, treat the file as housekeeping and measure visibility directly.
2. Using it to allow or block AI crawlers. It "controls nothing and blocks nothing" (opens in a new tab). Instead, use robots.txt, whose rules are standardized in RFC 9309 (opens in a new tab).
3. Letting it drift from the site. An old price in the file contradicts the page. Instead, update the file in the same edit as the page.
4. Writing instructions for AI into the file. Google's spam policies cover attempts to manipulate generative AI responses (opens in a new tab). Instead, list plain links and descriptions. Hidden instructions are covered in generative engine optimization antipatterns.
5. Publishing it and never linking or checking it. Agents fetch the file when directed (opens in a new tab). Instead, link it and check the logs.
6. Buying llms.txt as a visibility service. Google says third-party tools can't guarantee performance (opens in a new tab). Instead, ask for evidence that the file changed citations for anyone.

Checklist

FAQ

What is llms.txt?

A proposed Markdown file at /llms.txt that summarizes a website for language models and AI agents and links to its key pages. It was first proposed in 2024 (opens in a new tab), and Google Search does not use it (opens in a new tab).

Do we need an llms.txt file?

Most sites do not. Google says Google Search ignores the files (opens in a new tab), and 97% of published files got no requests in one study (opens in a new tab). A file makes sense if AI coding tools read your documentation, or if a partner such as Stripe's Directory (opens in a new tab) asks for one. If you keep one, keep it accurate.

Does ChatGPT read llms.txt?

Its search crawler rarely does. In one log study, OAI-SearchBot, PerplexityBot, and Claude's search crawler together made only a couple of hundred fetches across thousands of sites (opens in a new tab). OpenAI publishes an llms.txt for its own documentation, but its crawler documentation does not ask site owners to publish one (opens in a new tab).

Does Claude read llms.txt?

Claude's search crawler rarely does. In the same study, it was one of three AI search crawlers that together made only a couple of hundred fetches, while Anthropic's coding agent, Claude-Code, fetched the files more often than any AI search crawler or assistant (opens in a new tab). See how to rank in Claude for how Claude finds pages.

Does Google use llms.txt for AI Overviews?

No. Google says you don't need AI text files to appear in Google Search, including its generative AI features, "as Google Search itself doesn't use them" (opens in a new tab). What does matter for AI Overviews is covered in that guide.

Is llms.txt the same as robots.txt?

No. robots.txt tells crawlers which pages they may fetch (opens in a new tab). llms.txt blocks nothing (opens in a new tab); it only describes and links.

What is llms-full.txt?

A larger companion file that some documentation platforms generate. Mintlify, for example, describes it as a file that combines an entire documentation site into a single file, with each page's full Markdown content (opens in a new tab). The current proposal describes llms.txt and Markdown copies of single pages, not an llms-full.txt file. We found no data on who reads llms-full.txt files.

Is llms.txt like schema markup?

They solve different problems, but the evidence looks alike: neither has a measured effect on AI citations. See does schema markup help AEO? and what replicates in AEO and GEO research.

Change log

• September 28, 2026: First published.

Next steps

AEO HQ's Instant AEO Audit ($499) checks whether a site serves an llms.txt file, along with its robots.txt rules for AI crawlers, its sitemap, and the structured data types on its home page. Its limits are listed in our methodology.