
What Is llms.txt?
llms.txt is a proposed file, written in Markdown and placed on a website, that gives AI systems a short, curated guide to the site’s most important content. It was proposed in September 2024 by Jeremy Howard. It is not an official web standard, and Google says no such file is needed for its AI features.
| Question | Answer |
|---|---|
| What is it? | A Markdown file summarising a site and linking to its key pages |
| Who proposed it? | Jeremy Howard, September 2024 |
| Where does it go? | At /llms.txt, at the site root or a subpath |
| Is it a standard? | No; a community proposal |
| Does Google require it? | No |
| Is it the same as robots.txt? | No; it gives context, not access rules |
| Should you add one? | Useful for documentation-heavy sites; optional for others |
The description follows the proposal published at llmstxt.org.
Why Was llms.txt Proposed?
It was proposed to solve a practical problem. Language models work within a limited context window. A website’s HTML is full of navigation, scripts and advertising, which makes it hard for a model to find the essential information quickly.
The proposal’s answer is a single, clean file that says: here is what this site is, and here are the pages worth reading.
The authors expected it to be used mainly at inference time, when an AI agent is helping a user with a task, and not primarily for training models.
It appeals most to sites with technical documentation, where a developer might ask an AI assistant how to use a library or an API.
What Does the File Contain?
The file is plain Markdown, readable by people and programs. The proposal defines this order:
- An H1 with the name of the site or project. This is the only required part.
- A blockquote with a short summary.
- Optional detail in ordinary paragraphs or lists, with no headings.
- H2 sections containing lists of links, each in the form of a linked name, optionally followed by a colon and a note.
A section titled “Optional” has a special meaning: the links in it are secondary and can be skipped when a shorter context is needed.
An outline, described in words:
- A first line with a heading: the company name.
- A quoted line: one sentence on what the company does.
- A heading “Docs”, followed by links to the main guides, each with a short note.
- A heading “Products”, with links to product pages.
- A heading “Optional”, with links to background material.
The proposal also suggests offering Markdown versions of pages, at the same URL with “.md” added.
Where Should the File Be Placed?
At the path /llms.txt on your site, for example https://www.example.com/llms.txt. The proposal allows it at the root or at a subpath, where it covers the URLs beneath it.
- Serve it as plain text or Markdown.
- Keep it publicly accessible.
- Use full URLs in the links.
- Keep it short; it is a curated overview, not a full index.
Is llms.txt the Same as robots.txt?
No. They do different jobs.
| robots.txt | sitemap.xml | llms.txt | |
|---|---|---|---|
| Purpose | States which crawlers may access which URLs | Lists pages for indexing | Gives AI systems a curated overview |
| Format | Plain text directives | XML | Markdown |
| Audience | Crawlers | Search engines | Language models and agents |
| Status | Long-established standard | Long-established standard | Proposal |
| Controls access? | Yes | No | No |
| Required? | No, but widely expected | No, but recommended | No |
llms.txt cannot block or allow anything. To control whether AI crawlers may fetch your pages, use robots.txt. See our guide to robots.txt optimization.
Want your brand cited in AI answers?
Our AI search optimization work covers content structure, schema and E-E-A-T signals, measured every month.
See AI search optimizationWhich Tools Read llms.txt Today?
Adoption is partial and changing.
- Google: its documentation on AI features says you do not need to create new machine-readable files or AI text files to appear in AI Overviews or AI Mode.
- Other AI search engines: their published guidance focuses on crawler access through robots.txt. OpenAI’s help page on ChatGPT search and Perplexity’s crawler documentation do not list llms.txt as a requirement.
- Developer tools and AI coding assistants: some can be pointed at an llms.txt file to load documentation.
- Documentation platforms: several generate the file automatically.
- Individual users: can paste the file’s contents into an assistant.
The honest summary: there is no evidence that the major AI search engines use llms.txt to choose or rank sources. Its confirmed uses are in developer tooling and agent workflows. Check current documentation, since this may change.
Should You Add One?
It depends on your site.
| Type of site | Recommendation |
|---|---|
| Software with documentation or an API | Yes; real, practical uses exist |
| SaaS product with help content | Worth adding |
| Content-heavy publisher | Optional; low impact expected |
| Local business or small brochure site | Low priority |
| Ecommerce store | Low priority; product feeds and structured data matter more |
Arguments for:
- It takes little time.
- It carries almost no risk.
- It may help developer tools and agents use your content accurately.
Arguments against:
- No confirmed effect on visibility in major AI search engines.
- It needs maintenance; a stale file misleads.
- It can distract from work that certainly matters.
If your crawl access, indexing and content structure are not yet right, fix those first.
How Do You Create One?
- List your most important pages: what a newcomer must read to understand your product or organisation.
- Write a one-sentence summary of who you are.
- Group links under a few headings.
- Add a short note to each link saying what it covers.
- Put secondary material under “Optional”.
- Save the file as llms.txt, and upload it to the site root.
- Test that it loads at /llms.txt.
- Review it whenever key pages change.
Keep it honest and factual. Describe pages as they are.
What Should You Not Do?
- Do not stuff it with keywords. It is a guide, not a ranking lever.
- Do not include instructions aimed at manipulating models. That is deceptive.
- Do not list every URL. That is a sitemap’s job.
- Do not link to pages you block in robots.txt.
- Do not expect it to control access or training use.
- Do not let it replace clear, well-structured pages.
- Do not buy it as a service promising AI rankings.
What Matters More for AI Visibility?
Several things have documented effects, and should come before llms.txt.
- Crawler access. Allow the AI search crawlers you want, in robots.txt and at your firewall.
- Indexing on Google and Bing.
- HTML content. Many AI crawlers do not run JavaScript.
- Direct answers under clear headings.
- Accurate, consistent facts about your brand.
- Structured data that matches the visible page.
- Mentions on reputable sites.
Our guide to generative engine optimization covers each. For measuring results, see AI visibility tracking.
What Is the Sensible Position?
Treat llms.txt as a small, optional convenience. If you publish documentation, add it. If you run a typical business site, it will not hurt, and it should sit low on the list.
Do not believe claims that it is required, and do not believe claims that it is a secret advantage.
Frequently Asked Questions
Is llms.txt an official standard?
No. It is a community proposal published in September 2024, not a standard ratified by a standards body. Adoption is voluntary and uneven. Google states that no AI text files are needed for its AI features. Treat it as optional.
Where should the llms.txt file be placed?
At the path /llms.txt, normally at the root of your site, such as https://www.example.com/llms.txt. The proposal also allows it at a subpath, where it applies to the URLs beneath. It should be publicly accessible, written in Markdown and kept reasonably short.
Is llms.txt the same as robots.txt?
No. robots.txt tells crawlers which URLs they may access. llms.txt offers AI systems a curated summary of a site with links to key pages. It cannot allow or block anything. To control AI crawler access, you still need robots.txt, plus firewall rules where relevant.
Will llms.txt improve my visibility in ChatGPT or Google?
There is no confirmed evidence that major AI search engines use it to choose sources. Google says such files are not needed. Its practical uses today are mainly in developer tools and agent workflows. Crawler access, indexing and clear content matter far more.
How often should I update llms.txt?
Whenever the pages it lists change: new products, renamed documentation or removed sections. A file that points to outdated or missing pages misleads any system that reads it. If you cannot commit to maintaining it, it is better not to publish one.
Related Guides
Want this done for you?
See what each monthly plan includes, or send us your site and questions on WhatsApp. We do not promise rankings; we show the work and the data every month.
View SEO packages Chat on WhatsAppSources



