What is llms.txt, and does your site need one?
llms.txt is a Markdown file at the root of your site that gives AI models a short summary and a list of links to your most useful pages. It is cheap to add and does no harm, but no major AI provider’s crawler docs say they read it, and Google says Search ignores it.
Proposed by Jeremy Howard on 3 September 2024. An H1 with the site or project name is the only required section.
How this site makes its own
My /llms.txt is not a file I edit. It is built from the same data as the pages: the prices and every offer, then every live service page and guide with its description, grouped by topic. Drafts are left out, so it never links a page that does not exist. Terms, refunds and privacy go under the Optional heading the proposal suggests.
Because it is generated, it changes the moment a price or a page changes. A hand-written llms.txt goes out of date the first time someone forgets it.
What the llms.txt proposal says
- The file lives at /llms.txt, or under a path to cover only that part of a site.
- An H1 with the name of the site. This is the only required part, so the bar is low.
- A blockquote with a short summary.
- Optional paragraphs or lists with more detail, without headings.
- H2 sections that list links in Markdown, each with an optional note after a colon.
- A section called Optional for secondary links an AI can skip when it needs less text.
- Pages can also offer a clean Markdown copy at the same URL with .md added.
Who actually reads llms.txt
Here the evidence gets thin. OpenAI and Perplexity publish an llms.txt for their own developer docs, and llmstxt.org says Anthropic and Google’s Gemini docs do too. Publishing one is not the same as reading yours.
I opened the crawler pages of OpenAI, Anthropic and Perplexity again on 17 November 2026. None of them says their bots read llms.txt on other people’s sites. Google is the most direct: its AI optimization guide says you do not need machine-readable or AI text files to appear in Google Search, and that keeping them for other services will neither harm nor help, because Google Search ignores them.
One tool does check for it. llmstxt.org says Chrome’s Lighthouse audits sites for an llms.txt in its agentic browsing checks, which test how well AI agents can use a site.
When an llms.txt is worth it
- Developer docs and APIs, where people paste your docs into a coding assistant.
- Sites with many pages, where a short map helps an agent find the right one.
- When it can be generated, so nobody has to remember it exists.
When not to spend time on it
If your site cannot be crawled, renders its content only with JavaScript, or blocks AI search bots in robots.txt, fix those first. They decide whether assistants can see you at all, and no text file will talk a bot past a firewall.
To make one by hand, the free llms.txt generator at /tools/llms-txt-generator builds a file in the format above.
Read next
Sources
- llmstxt.org, the /llms.txt file proposal, modified 10 Aug 2026
- Google Search Central, AI optimization guide, updated 10 Jul 2026
- Google Search Central, documentation updates (llms.txt note, 15 Jun 2026), seen 17 Nov 2026
- OpenAI, overview of OpenAI crawlers, seen 17 Nov 2026
- Anthropic, how site owners can block the crawler, updated 4 Nov 2026
- Perplexity, Perplexity crawlers, seen 17 Nov 2026