llms.txt: The Complete, Evidence-Aware Guide

llms.txt is a useful experiment when it is described accurately. It is a community proposal for a concise Markdown guide to a website. It is not an access-control standard, and there is no general evidence that publishing the file increases citations across AI platforms.

That boundary makes the implementation decision straightforward: publish a small, accurate file if maintaining it is cheap, but do not treat it as a substitute for robots.txt, a sitemap, internal links, or good pages.

What the proposal defines

The llms.txt proposal describes a Markdown file at /llms.txt with:

• an H1 containing the project or site name;
• a short summary;
• optional explanatory text;
• H2 sections containing curated Markdown links;
• an optional section for secondary resources.

A minimal example:

# Example Company

Example Company provides commercial energy audits in New Zealand.

## Services

- [Commercial energy audits](https://example.com/audits/): Scope, process, and deliverables.
- [Monitoring](https://example.com/monitoring/): Ongoing energy-use reporting.

## Guides

- [Preparing for an audit](https://example.com/audit-preparation/): Information to collect before the site visit.

This is a curated guide, not a sitemap dump. Descriptions should state what a reader will find rather than repeat keywords.

What Google says

Google’s current guide to generative AI features explicitly addresses the format: Google Search ignores llms.txt, and maintaining one neither helps nor harms visibility or rankings in Google Search.

Google also says there is no new machine-readable AI file required for AI Overviews or AI Mode. Normal Search requirements still apply: crawl access, indexing eligibility, useful text, internal discoverability, and compliance with Search policies.

Therefore, do not sell llms.txt as a Google AI Overview tactic.

What other platforms document

Some developer documentation sites publish their own llms.txt files, and some tools may consume them. Publication is not the same as a platform-wide commitment to use the file for public search selection.

OpenAI’s public publisher guidance tells sites that want inclusion in ChatGPT summaries and snippets not to block OAI-SearchBot. Perplexity’s crawler documentation gives corresponding guidance for PerplexityBot. Neither document makes llms.txt a prerequisite for inclusion.

When support matters, look for current platform documentation or controlled server-log evidence. Do not infer crawler use from the mere existence of a file.

llms.txt, robots.txt, and sitemap.xml

Use robots.txt for access choices. Use a sitemap for canonical discovery. Treat llms.txt as an optional editorial layer.

How to create a useful file

1. Define the audience and purpose

Write one sentence that states what the organization does, who it serves, and where it operates when location matters.

2. Select canonical pages

Include only pages that help a reader understand the business or project:

• primary services or products;
• authoritative documentation;
• strong guides or case studies;
• useful tools;
• essential identity or contact pages.

Exclude login, cart, filter, duplicate, thin, staging, and non-canonical URLs.

3. Write factual descriptions

Each description should explain scope. Avoid claims such as “best,” “leading,” or “AI optimized” unless the page proves them.

4. Validate the response

Check that:

• /llms.txt returns 200;
• the response is readable plain text or Markdown;
• every linked URL is absolute, canonical, and live;
• redirects do not point to retired hosts;
• the file contains no secrets or private URLs;
• changes to key routes update the file.

Our llms.txt generator can create the initial structure, and the llms.txt checker can inspect a published file. Human review is still necessary because a syntactically valid list can be stale or badly curated.

Measuring whether it is used

Do not measure success by the file’s existence. If a system claims support, test it:

1. record the platform, version or mode, date, and query;
2. publish a unique but non-sensitive change in the file;
3. inspect verified crawler logs where possible;
4. compare repeated results while holding other variables steady;
5. avoid claiming causation when the same pages are discoverable elsewhere.

Even a verified fetch proves that the system requested the file, not that the file changed citation selection.

When the file is worth maintaining

Publish llms.txt when your site already has strong technical foundations and the file can be generated or curated without drift. Skip or postpone it when the site still has blocked pages, broken canonicals, orphaned content, an inaccurate sitemap, or weak primary pages.

The right priority order is usually:

1. crawl and indexing controls;
2. accurate, useful content;
3. canonical URLs and internal links;
4. structured data that matches visible content;
5. measurement;
6. optional experimental discovery files.

For the surrounding work, see the GEO guide and the article on measuring AI visibility. The evidence-aware conclusion is modest: llms.txt can be a clean, low-cost site guide, but platform support and business impact must be demonstrated rather than assumed.