The Complete llms.txt Guide — Authoring, Operations, and Whether It Works
What llms.txt and llms-full.txt are, why they were proposed, and how to author and maintain them, with worked examples. Covers how they differ from robots.txt, the debate over whether they actually work, and a practical checklist.
llms.txt is a Markdown file placed at the site root. The proposal is to give large language models (LLMs) a tidy list of links so they can quickly find a site's most important pages. It was proposed in 2024 at llmstxt.org.[1] The name evokes robots.txt and sitemap.xml, but its purpose and behavior differ from both. This article lays out the problem llms.txt sets out to solve, how to write one, how it relates to robots.txt, and the debate over whether it actually works.
Why llms.txt was proposed — the problem and the fix
When an LLM reads a web page, it runs into two problems. First, an ordinary HTML page mixes navigation bars, ads, cookie banners, and JavaScript into the content. Second, an LLM's context window is finite, so the more noise there is, the less room is left for the content that actually matters. The cause is the structural mismatch of "a machine reading, as-is, a page built for humans," and the effect shows up as the LLM missing the core point or citing the wrong part.
The llms.txt fix is simple: gather, in one place, a noise-free Markdown list saying "here are the important pages on this site." Where possible, it recommends also providing each link target in clean Markdown (.md) form. The idea is to separate the flashy page humans see from the tidy, machine-readable text. The thinking behind this approach connects directly to the "make it readable for AI" stage covered in the GEO/AEO technical-axis methodology.
How to write one, with an example
The format llmstxt.org proposes simply follows Markdown heading rules.[1] At the top, write the site or project name as an H1, and add a one-line summary as a blockquote (>). Then, under each H2 section, group the related links as a list. The link format is [title](URL): short description. Adding a final ## Optional section lets you mark supplementary links that can be skipped when the context budget is tight.
# Company Help Center
> A collection of guides covering how to use the product and the API docs.
## Documentation
- [Getting started](https://example.com/start.md): Installation and first setup
- [API reference](https://example.com/api.md): The full list of endpoints
## Optional
- [Changelog](https://example.com/changelog.md): Changes by version
The variant that places the full content directly in a single file, rather than beyond a link, is llms-full.txt. The intent is to let an LLM read the content right away, without the extra requests of following links. When there are many documents, the file can grow large and actually eat into the context, so llms.txt (the table-of-contents form) and llms-full.txt (the full-text form) are usually published together.
| Aspect | llms.txt | llms-full.txt |
|---|---|---|
| What it holds | A list of links to key pages | The full content text |
| Size | Small | Grows in proportion to the number of documents |
| Purpose | Points out what matters | Lets the content be read directly |
| Location | /llms.txt | /llms-full.txt |
Both files go at the site root (https://domain/llms.txt). They must sit at the root rather than a subpath so that tools find them by convention.
How it relates to robots.txt and sitemap
The names are similar, so they are easy to confuse, but their roles differ. robots.txt is a rules file that allows or blocks crawler access.[2] If you want to block AI crawlers, you specify and block user agents like GPTBot or Google-Extended in robots.txt.[3][4] sitemap.xml is a machine-readable list that enumerates every URL to be indexed, leaving none out.
llms.txt replaces neither of these. It does not control access, nor does it enumerate every URL. Instead, it curates: "among your many pages, this is what an LLM should read first." So the three files are not in competition but a division of labor. Set your access policy with robots.txt, announce your indexing scope with the sitemap, and point to the essentials with llms.txt. For a deeper treatment of blocking and allowing AI crawlers, see AI crawler management; for machine-readable structured data, see structured data and schema.
The debate over whether it works, and its limits
The biggest point of contention around llms.txt is "does anything actually read it?" As of 2026, there is no clear case of a major AI engine confirming that it officially reads llms.txt and reflects it in answers. llms.txt is, at most, a community proposal and not an official standard. Some developer tools and documentation sites have adopted it voluntarily, but adoption does not guarantee consumption by an engine.
Even so, the case for it being worth creating rests on its low cost relative to risk. Because it is just adding a single file, there is almost no downside; it has the side benefit of tidying up the site's core structure; and if engines support it down the road, you benefit immediately. Conversely, overconfidence is a mistake. llms.txt is not a device that generates citations directly, but an aid that helps an LLM read your content efficiently. Citations, in the end, are the combined result of your content's structure and credibility together with crawler accessibility.
Practical checks
- Verify the root path: check that it opens directly at
domain/llms.txt. If it sits on a subpath, tools will not find it. - Readability of link targets: where possible, keep link targets in Markdown (.md) or on pages with clearly legible content.
- Curation: do not include every page — pick only the essentials an LLM should read first. The goal is removing noise.
- Audit robots.txt: blocking AI crawlers while publishing only llms.txt is pointless. Keep your access policy and your guidance aligned.
- Updates: when documents change, update llms.txt too. Dead links actually undermine trust.
Wrap-up
llms.txt is a Markdown guide file that helps an LLM efficiently read a site's essentials. Its role does not overlap with robots.txt (access control) or the sitemap (indexing list), and there is also the llms-full.txt variant that holds the content in its entirety. Because it is still a proposal — with no official standard or universal engine support confirmed yet — it is no silver bullet, but its added cost is low and it has a site-tidying benefit, so it is worth trying. The key is not to overrely on it, while tidying up your access policy and content structure together.
Frequently asked questions
- No, it is not mandatory. llms.txt is a proposal rather than a standard, and no major AI engine has confirmed that it officially reads the file yet. That said, the cost of creating one is low, and tidying up your site structure helps you prepare for future support, so it is worth a try.
- robots.txt is a rules file that allows or blocks crawler access, while llms.txt does not control access at all. llms.txt is a kind of guide that tells an LLM which pages matter. If you want to block AI crawlers, use robots.txt; if you want to point to important content, use llms.txt. The two do not conflict and are used together.
- llms.txt is a list of links pointing to your key pages. llms-full.txt is a variant that gathers that content as actual text in a single file, letting an LLM read the content directly without following links. When you have many documents, llms-full.txt can grow large, so the two are often published together.
- It goes directly under the root domain — that is, at the path https://example.com/llms.txt. It follows the same location rule as robots.txt. It must sit at the root rather than a subpath so that tools and crawlers can find it by convention.
- It does not guarantee a direct causal effect. llms.txt is an aid that helps an LLM read your content more efficiently, not a device that generates citations on its own. Citations are the combined result of your content's structure and credibility together with crawler accessibility.
Q.Do I have to create an llms.txt file?
Q.How does llms.txt differ from robots.txt?
Q.How does llms.txt differ from llms-full.txt?
Q.Where should llms.txt go?
Q.Does publishing llms.txt increase AI citations?
Sources
Related documents
- Structured Data and Schema Guide for AEOStructured data (JSON-LD) from schema.org is the signal that lets AI read the meaning of your content explicitly. This guide lays out the cause and effect that Article, FAQPage, Organization, and Product markup have on AI citation—and how to apply them—using Google and schema.org sources with JSON-LD examples.
- Managing AI Crawlers — Allowing GPTBot, ClaudeBot, and PerplexityBot and the Trade-offsHow to identify GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Google-Extended, and the visibility trade-offs of allowing or blocking them in robots.txt — based on OpenAI's and Google's official documentation.
- Tech GEO + Content GEO — A Two-Axis Method Linking Diagnosis and CreationA framework that splits GEO into a technical diagnostic axis (Technical GEO) and a content creation axis (Content GEO). It lays out what each axis checks and executes, and how the two connect, with a side-by-side comparison table.