llms.txt

TL;DR — llms.txt is an emerging standard (llmstxt.org, 2024) analogous to robots.txt but specifically for AI crawlers. It provides a curated, LLM-friendly summary of your site in Markdown. Perplexity already supports it. It's a 30-minute implementation with long-term AI discoverability upside.

Why llms.txt Matters for AI Engines

The llms.txt specification (introduced at llmstxt.org in 2024) addresses a fundamental problem: AI engines that crawl the web to answer queries receive all content equally, without knowing which pages are most important or authoritative for a given site. The llms.txt file solves this by providing a curated, LLM-readable index of your site's most important content.

Unlike robots.txt, which tells crawlers what not to access, llms.txt tells AI engines what is most important and how to understand your site. It's a structured summary in plain Markdown format — site name, description, and a list of key pages with brief descriptions.

The spec also defines llms-full.txt — a companion file containing the complete content of your key pages in a single, LLM-consumable document. This is particularly valuable for AI engines that build knowledge graphs of sites: rather than crawling 500 pages, they can read one file that contains everything important.

Perplexity announced support for llms.txt in early 2025. As the spec gains adoption, AI engines that support it will have a more accurate, curated understanding of your site compared to AI engines that have to infer structure from crawling.

How to Implement

Create /llms.txt at your site root, accessible at https://yourdomain.com/llms.txt:

# Your Company Name

> One paragraph description of your site: what it does, who it's for, and what makes it authoritative.

## Key Pages

- [Home](https://yourdomain.com/): Main landing page — overview of what we do
- [Product](https://yourdomain.com/product): Full product description and features
- [Blog](https://yourdomain.com/blog): Articles and guides on [your topic]
- [About](https://yourdomain.com/about): Team, mission, and company background

Optionally, create /llms-full.txt with the complete text content of your most important pages.

Common Mistakes

• Using HTML instead of plain Markdown — the spec explicitly requires plain Markdown; HTML tags will confuse LLM parsers
• Not updating when major pages are added — llms.txt should reflect your current site structure; treat it like a sitemap
• Including pages you don't want AI engines to use — llms.txt is a positive curation tool; only include pages you actively want AI engines to read and cite

Sources

• llmstxt.org specification
• Anthropic crawl policy
• OpenAI GPTBot documentation