A new file has started appearing on websites alongside robots.txt and sitemap.xml: llms.txt. It has generated plenty of discussion in SEO circles, with some calling it essential for AI visibility and others dismissing it as unnecessary. The truth is more nuanced, and it depends on what kind of website you run.
This guide explains what llms.txt is, how it works, what it can and cannot do, and whether you should create one.
What Is llms.txt?
llms.txt is a proposed standard for a plain-text file, written in Markdown, placed at the root of a website, such as https://example.com/llms.txt. Its purpose is to give large language models a concise, curated overview of a site’s most important content, with links to key pages or LLM-friendly versions of documentation.
The proposal was published in 2024 by Jeremy Howard of Answer.AI. It was designed mainly to help language models use website content at inference time, meaning when a model or AI tool is answering a question, particularly for technical documentation.
What Does an llms.txt File Look Like?
A typical llms.txt file follows a simple Markdown structure:
# Example Company
> Example Company provides project management software for construction teams.
## Documentation
- [Getting started](https://example.com/docs/start): Setup guide for new users
- [API reference](https://example.com/docs/api): Endpoints and authentication
## Company
- [About us](https://example.com/about): Company overview and history
- [Pricing](https://example.com/pricing): Plans and featuresThe format includes a title, a short summary, and sections with links and brief descriptions. Some sites also provide an expanded file, often called llms-full.txt, containing more complete content.
llms.txt vs robots.txt vs sitemap.xml
| File | Purpose | Audience |
|---|---|---|
| robots.txt | Tells crawlers which areas they may or may not access | Crawlers, including search and AI bots |
| sitemap.xml | Lists URLs for search engines to discover and crawl | Search engines |
| llms.txt | Offers a curated summary and links for language models | LLMs and AI tools, if they choose to use it |
An important distinction: llms.txt does not control access. It does not block or allow crawlers. That remains the job of robots.txt and server-level controls.
Do AI Search Engines Use llms.txt?
This is the key question. At the time of writing, major AI search providers have not publicly confirmed that they use llms.txt as a signal for retrieval, ranking, or citations. Google representatives have indicated that Google Search does not use it. Some AI developer tools and documentation platforms support it, and some companies publish llms.txt files for their own documentation.
In practice, AI search engines still rely primarily on crawling your actual pages, search indexes, and the signals discussed throughout our guide to how AI search works. Creating llms.txt is unlikely to improve your AI search visibility on its own.
When llms.txt Can Be Useful
Developer Documentation
If you run a software product with extensive documentation, llms.txt can help developers and AI coding tools load a concise map of your docs. This is where the proposal has seen the most adoption.
Complex Sites With Deep Content
Sites with large knowledge bases may benefit from a curated overview that points AI tools to the most important resources.
Future-Proofing
Creating an llms.txt file is low effort. If adoption grows, you will already have one. Just do not expect immediate results.
When It Is Probably Not a Priority
For most small business, local service, and ecommerce websites, llms.txt offers little measurable benefit today. Your time is better spent on:
- Allowing AI search crawlers in robots.txt
- Making content render in HTML
- Writing answer-first, evidence-rich content
- Strengthening entity and brand signals
- Earning third-party mentions and reviews
These have a far more direct impact on AI visibility. See our AI SEO checklist for priorities.
How to Create an llms.txt File
If you decide to create one:
- Write a concise title and summary describing your organisation or product.
- List your most important pages grouped into logical sections.
- Add a short description for each link explaining what it covers.
- Keep it curated. Include key resources, not every URL.
- Upload it to your root directory so it is accessible at /llms.txt.
- Keep it updated when important pages change.
On WordPress, you can upload the file via your hosting file manager or use a plugin that supports it. Make sure the file is publicly accessible.
Common Misconceptions
- “llms.txt blocks AI training.” It does not. Use robots.txt rules for crawlers such as GPTBot or Google-Extended.
- “llms.txt guarantees AI citations.” There is no evidence of this.
- “Every site needs llms.txt.” It is optional and most useful for documentation-heavy sites.
- “llms.txt replaces sitemaps.” Sitemaps remain important for search engines.
Final Thoughts
llms.txt is an interesting proposal that may become more useful as AI tools evolve. For documentation-heavy sites, it can be a helpful addition. For most businesses, it is a nice-to-have rather than a priority. Focus first on the fundamentals that clearly influence AI visibility: crawler access, clear content, strong evidence, and consistent brand signals. For a full technical review, consider an AI SEO audit, and for broader context, see our guide to AI SEO.
Frequently Asked Questions
What is llms.txt?
llms.txt is a proposed Markdown file placed at a website’s root that gives language models a concise, curated overview of the site’s most important content, with links and short descriptions.
Does llms.txt improve AI search visibility?
There is no confirmed evidence that major AI search engines use llms.txt for retrieval or citations. Crawler access, clear content, evidence, and brand signals have far more direct impact.
Is llms.txt the same as robots.txt?
No. robots.txt controls which areas crawlers may access. llms.txt does not block or allow anything; it simply offers a curated summary and links that AI tools may choose to use.
Can llms.txt stop AI models training on my content?
No. To manage AI training access, use robots.txt rules for training-related user agents such as GPTBot or Google-Extended, and consider additional server-level or firewall controls where needed.
Who should create an llms.txt file?
Sites with extensive developer documentation or large knowledge bases benefit most. For most small business and ecommerce sites, it is optional and lower priority than core AI SEO work.
How do I add llms.txt to WordPress?
Create a Markdown text file with a title, summary, and curated links, then upload it to your site’s root directory via hosting file manager or a supporting plugin, and confirm it loads publicly.
