Robots.txt for AI Search: Every AI Crawler and What It Controls

Robots.txt controls AI crawlers one token at a time, and each token does a different job. GPTBot and ClaudeBot collect training data, OAI-SearchBot and PerplexityBot feed AI search, ChatGPT-User fetches a page when someone asks, and Google’s AI Overviews ride on plain Googlebot. We checked every token below at the vendor’s own documentation on 11 November 2026, and our own robots.txt lets every one of them in.

Your robots.txt is a few kilobytes of plain text. Martijn Koster wrote the original rules in 1994, and for most of the years since, the file talked to search engines. Set it and forget it.

That era is over. OpenAI, Anthropic, Google, Perplexity, Apple, Amazon, Meta and Microsoft all publish robots.txt controls for AI, and most of them document more than one token. If your robots.txt still looks like it did in 2023, you’re either missing from AI answers or handing out access you never decided on. Both are easy to fix.

Below: what each token controls, what we allow on webaloha.co and why, copy-ready rules for the common goals, and how to test the result. Syntax basics live in our guide on how to check robots.txt, and WordPress owners can jump to editing robots.txt in WordPress.

Training, search and user fetches: three kinds of AI crawler

Not every AI bot does the same thing. Get this one idea right and the rest is easy. The big AI companies split their bots by job:

• Training crawlers collect pages that may be used to train future models. Block them and you opt out of training. Examples: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended.
• Search crawlers build the index that AI search answers come from. Block them and you drop out of that platform’s answers. Examples: OAI-SearchBot, Claude-SearchBot, PerplexityBot, and for Google, Googlebot itself.
• User-triggered fetchers open one page because a person asked. Several vendors say robots.txt may not apply to them. Examples: ChatGPT-User, Claude-User, Perplexity-User.

The trap most guides miss: “block AI” is not one decision. Blocking GPTBot and blocking OAI-SearchBot do completely different things, even though both say OpenAI on the label.

Platform controls at a glance

Sources, all checked on 11 November 2026: Google common crawlers, Google AI features, OpenAI crawlers, Anthropic crawlers, Perplexity crawlers, Apple’s Applebot page, Amazonbot, Meta web crawlers, Common Crawl’s CCBot page and Bing’s October 2026 announcement.

Vendors add and rename bots without much fanfare. If a token isn’t in this table, we couldn’t find it in the vendor’s own documentation today.

What our own robots.txt allows, and why

Our robots.txt comes from a small route in our Astro site, not a hand-edited file. This is what webaloha.co served on 11 November 2026, trimmed where the same three lines repeat:

# Web Aloha robots.txt
# Crawl the public site freely.
# API responses send X-Robots-Tag: noindex, nofollow. They remain crawlable
# so search engines can observe that directive; they are not sitemap URLs.
# Content use: allow search, AI input, and AI training.

User-agent: *
Content-Signal: ai-train=yes, search=yes, ai-input=yes
Allow: /

User-agent: OAI-SearchBot
Content-Signal: ai-train=yes, search=yes, ai-input=yes
Allow: /

# ...the same three lines again for ChatGPT-User, GPTBot, ClaudeBot,
# anthropic-ai, PerplexityBot, Google-Extended, CCBot, Bytespider
# and Meta-ExternalAgent

Sitemap: https://webaloha.co/sitemap.xml

Translation: everyone is welcome, for everything. Search, AI answers and AI training. The Content-Signal line says the same thing in Cloudflare’s machine-readable format, which we explain further down.

Why so open? We sell web design, SEO and GEO work, and we publish 95 free tools and a stack of guides. We want ChatGPT, Claude, Perplexity and Gemini to know what we do and send people to those tools. Being in a training set costs us nothing we care about. Being missing from AI answers would.

How the file got here:

• March 2026: the first version welcomed the named AI crawlers and blocked /api/ for everyone.
• July 2026: we added the Content-Signal line.
• August 2026: we removed Disallow: /api/. Our API responses send X-Robots-Tag: noindex, nofollow (we rechecked the live header today), and a crawler that is blocked from a URL never gets to read that header. Blocking the crawl would have hidden the very signal that keeps those URLs out of search.

One honest leftover: while rechecking every token for this update, we noticed our file still names anthropic-ai, which Anthropic’s crawler page doesn’t list today, and doesn’t name Claude-SearchBot or Claude-User. No harm done, because both fall through to the * group, which allows everything. It’s still a good reminder of how quietly a robots.txt goes stale.

Did the open door get us cited? Sometimes. On 26 September 2026 we asked six questions in logged-out ChatGPT, Perplexity and Google AI Mode. Perplexity mentioned Web Aloha in 2 of 6 answers. ChatGPT mentioned us once, when the question named us. Google AI Mode showed a bot check, so we couldn’t measure it. Allowing crawlers got us into the room; it didn’t get us quoted. The full snapshot sits on our GEO services page.

Do not copy a universal allow list

Our policy fits us. It won’t fit everyone. Before you copy anyone’s robots.txt, ours included, answer these:

1. Is the content public, licensed, paywalled, private or regulated?
2. Is the goal search discovery, referrals, a training opt-out or less server load?
3. Which token does the vendor tie to that goal? The table above has them.
4. Does a CDN or firewall already block crawlers that your robots.txt allows? Cloudflare’s AI crawler policies can.
5. Are there private URLs that need a login rather than a robots.txt line?

Our rule of thumb hasn’t changed since the first version of this article:

• You sell products or services: allow everything. A local plumber, a SaaS company, a web shop: you want ChatGPT recommending you and Perplexity citing you. The downside of AI invisibility is bigger than the cost of being crawled.
• You publish premium or licensed content: block the training crawlers and keep the search crawlers open, so AI answers can still cite and link you.
• You have paid or private content: put it behind a login. RFC 9309, the robots.txt standard, says the rules “are not a form of access authorization”.

Ready-to-copy robots.txt rules for AI crawlers

Replace example.com, keep your existing rules for admin areas, and test before and after. Prefer clicking to typing? Our Robots.txt Generator toggles AI bots one by one. Mind its Block all AI button: it adds search and user-triggered tokens too, so it is broader than a training opt-out.

Allow every crawler, AI included

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

This is what most businesses need. An empty file, or a robots.txt URL that answers with a 404, lands in the same place: RFC 9309 lets crawlers access everything when the file returns a 4xx status. A 5xx is the opposite. If your robots.txt throws a server error, crawlers that follow the standard must assume the whole site is disallowed.

Allow AI search, block AI training

# Search engines, AI search and user fetches: welcome
User-agent: *
Allow: /

# Model training and training datasets: no thanks
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: CCBot
Disallow: /

# The shared AI-training preference: Bing reads it from
# 1 February 2027 and Google plans to support it in early 2027
User-agent: __usage_ai-training
Disallow: /

Sitemap: https://example.com/sitemap.xml

Several user agents can share one group of rules; RFC 9309 allows it. Three notes before you paste. meta-externalagent also covers Meta’s product indexing, and Amazonbot also improves Amazon’s products, while Amazon’s search crawler is Amzn-SearchBot. Google-Extended also covers grounding in Gemini Apps, so you step out of that too. And Google AI Overviews stay untouched, because they come from Googlebot.

Block AI crawlers, keep Google and Bing

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Meta-WebIndexer
User-agent: Meta-ExternalFetcher
User-agent: Amazonbot
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

This takes you out of ChatGPT search, Claude’s search index, Perplexity and Meta AI citations. It does not take you out of Google AI Overviews, which use Googlebot, or Bing, where AI use is set with Microsoft’s preferences rather than a separate bot. And the user-triggered fetchers on that list may ignore it, so a real block needs your firewall or CDN.

Block every crawler

User-agent: *
Disallow: /

Every crawler that follows the standard stops, Googlebot included. Fine for a staging site, and even there, add a password. Our robots.txt checking guide has more side-by-side examples.

A conservative public-site example

This example keeps public search crawlers open and blocks an internal area. It intentionally does not prescribe a training policy:

User-agent: Googlebot
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
Allow: /
Disallow: /internal/

User-agent: *
Disallow: /internal/

Sitemap: https://example.com/sitemap.xml

A crawler that finds a group naming it follows that group and ignores the * group, so the /internal/ rule is repeated in the named group. Only include /internal/ if that route exists, and still protect it with authentication if it contains anything private. One side effect worth knowing: Apple says Applebot follows your Googlebot rules when no group names Applebot, so the Googlebot group above applies to Apple’s search crawler too.

Google Search and Google-Extended are different controls

Here’s the part that surprises people: there is no AI Overviews crawler. Google’s AI features documentation says Googlebot’s robots.txt rules are the control for Search crawling, and that AI Overviews and AI Mode have no additional requirements. A page needs to be indexed and eligible to show a snippet. That’s it.

So you can’t block AI Overviews with robots.txt without blocking Google Search. If you want less of your text in AI features, Google points to nosnippet, data-nosnippet, max-snippet and noindex.

Google-Extended is a robots.txt token with no crawler behind it: Google says it has no separate user agent string, and the crawling is done by Google’s existing crawlers. It decides whether content Google crawls may train future Gemini models and be used for grounding in Gemini Apps and in Grounding with Google Search on Vertex AI. Google says it does not affect inclusion in Google Search and is not a ranking signal.

Therefore, this is a valid policy:

User-agent: Googlebot
Allow: /

User-agent: Google-Extended
Disallow: /

It keeps Google Search crawl access while declining the Google-Extended uses. Whether that’s right for you is a business and publishing call, not an SEO rule.

OpenAI search and training are separate

OpenAI documents four user agents, and says each setting is independent of the others:

• OAI-SearchBot surfaces websites in ChatGPT search. Sites that opt out are not shown in ChatGPT search answers, though they can still appear as navigational links. OpenAI says a robots.txt change can take about 24 hours to reach search results.
• GPTBot crawls content that may be used to train OpenAI’s foundation models. Disallowing it is your training opt-out.
• ChatGPT-User handles actions a person starts in ChatGPT or a custom GPT, such as opening a page to answer a question. It doesn’t crawl automatically and doesn’t decide search inclusion, and OpenAI says robots.txt rules may not apply to it.
• OAI-AdsBot only visits pages submitted as ads in ChatGPT, to check their safety, and OpenAI says that data isn’t used to train its foundation models.

For example, a publisher could allow ChatGPT search while opting out of training:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

If both are allowed, OpenAI says it may use one crawl for both purposes. If the intent is to keep a page out of an index rather than only stop crawling, remember that a crawler has to access a page to see a page-level noindex.

Anthropic: ClaudeBot, Claude-SearchBot and Claude-User

Anthropic documents three bots, each with its own robots.txt token:

• ClaudeBot collects web content that could contribute to training Claude models. Blocking it tells Anthropic to leave your future content out of its training data.
• Claude-SearchBot analyzes pages to make Claude’s search results more relevant and accurate. Blocking it keeps your pages out of that index, which can reduce your visibility in Claude’s answers.
• Claude-User fetches pages when a person asks Claude a question. Blocking it stops Claude retrieving your pages for those requests.

Anthropic says its bots honor robots.txt and support the non-standard Crawl-delay directive. The rules must sit in the robots.txt of each subdomain you want to cover. Anthropic warns that blocking its IP addresses may not work reliably, because the bots then can’t read your robots.txt at all; it publishes its crawler IPs at claude.com/crawling/bots.json for log checks.

New since 30 October 2026: Anthropic recognizes a machine-readable “no AI training” preference based on the IETF aipref draft. If your robots.txt names ClaudeBot, that rule wins over the general preference. The older tokens anthropic-ai and Claude-Web don’t appear on Anthropic’s page today.

To stay in Claude’s search answers while opting out of training:

User-agent: Claude-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

We checked these names against Anthropic’s documentation on 11 November 2026.

Perplexity has an automated crawler and a user fetcher

Perplexity documents two agents. PerplexityBot exists to surface and link websites in Perplexity search, and Perplexity recommends allowing it if you want to appear there. Perplexity says it is not used to crawl content for AI foundation models.

Perplexity-User is different: it fetches a page because a user asked a question, and Perplexity says it generally ignores robots.txt rules. Changes to your file can take up to 24 hours to show, and Perplexity publishes IP ranges for both agents at perplexity.ai/perplexitybot.json and perplexity.ai/perplexity-user.json.

This is why a robots audit shouldn’t treat every observed user agent as an equal opt-in switch. Some are automated crawlers; others stand in for a person.

Apple, Amazon, Meta, Common Crawl and Microsoft

Apple: Applebot and Applebot-Extended

Applebot powers search in Siri, Spotlight and Safari, and Apple says the data it crawls may also help train Apple’s foundation models. Applebot-Extended doesn’t crawl anything: it tells Apple not to use your content for training its general-purpose foundation models. Pages that disallow Applebot-Extended can still appear in Siri, Spotlight and Safari search. Blocking Applebot itself, or using noindex, is what keeps a page out of Spotlight and Siri Suggestions.

Watch the fallback: when no group names Applebot but one names Googlebot, Applebot follows the Googlebot rules.

Amazon: Amazonbot, Amzn-SearchBot and Amzn-User

Amazon documents three agents. Amazonbot improves Amazon’s products and services and may be used to train Amazon AI models. Amzn-SearchBot makes content eligible for search experiences in Amazon products, such as Alexa, and isn’t used for generative AI training; if your file doesn’t name it but allows other search bots, it follows their rules. Amzn-User handles user requests, such as Alexa questions that need current information, and may not follow every robots.txt rule. Amazon treats a page-level noarchive tag as a request not to use that page for model training.

Meta: meta-externalagent, Meta-WebIndexer and Meta-ExternalFetcher

Meta’s crawler page lists meta-externalagent for training foundation AI models and indexing content for Meta’s products, and Meta-WebIndexer for Meta AI search, where Meta says allowing it helps them cite and link your content. Meta-ExternalFetcher fetches links at a user’s request, including agent tasks, and may bypass robots.txt. Meta asks for up to 24 hours for robots.txt changes to take effect.

Matching on the user-agent name is case-insensitive under RFC 9309, so Meta-ExternalAgent in our file works the same as Meta’s lowercase spelling.

Common Crawl: CCBot

CCBot builds Common Crawl’s open repository of web crawl data, which anyone can download. To keep it out, Common Crawl gives the plain User-agent: CCBot and Disallow: / pair. It also warns that other crawlers pretend to be CCBot, so it publishes its IP ranges at index.commoncrawl.org/ccbot.json and supports reverse DNS checks.

Microsoft: Bingbot and the new AI preferences

Microsoft’s 30 October 2026 announcement names no separate AI crawler. Bingbot stays the crawler, and two content-use preferences arrive instead. AI training is set in robots.txt with the __usage_ai-training token, or per page. AI grounding, meaning content used to inform AI-generated answers, is set per page only, with a Bingbot meta tag or an X-Robots-Tag header such as ai-grounding:disallow. Bing starts honoring both on 1 February 2027, and after 1 March 2027 it stops reading noarchive and nocache as AI preferences. Both default to allow, and neither overrides a robots.txt rule that blocks Bingbot.

What about Bytespider?

ByteDance’s Bytespider shows up in plenty of robots.txt files, ours included. We couldn’t find documentation from ByteDance itself on 11 November 2026, so it isn’t in our table: we only list tokens their own vendor documents.

New opt-out signals: Content-Signal and AI training preferences

Two newer lines say how content may be used, not just who may crawl it.

Content-Signal. Cloudflare published the Content Signals Policy in September 2025 under CC0. A line such as Content-Signal: search=yes, ai-train=no sets three switches:

• search: building a search index and showing links and short excerpts. Cloudflare says this doesn’t include AI-generated search summaries.
• ai-input: feeding content into AI models when they answer, such as retrieval and grounding.
• ai-train: training or fine-tuning AI models.

A missing signal means nothing either way. Cloudflare calls these preferences, not technical countermeasures, and says some companies may ignore them. Our own file says yes to all three. If Search Console flags the line as “Syntax not understood”, Cloudflare’s documentation says that doesn’t affect crawling.

__usage_ai-training. This is the shared training preference from the IETF aipref work. Bing reads it from 1 February 2027, Google says it intends to support it in early 2027, and Anthropic has recognized an aipref-based “no AI training” signal since 30 October 2026. It costs one group of two lines, as in the training example above.

What robots.txt cannot do

Robots.txt is a polite sign on the door. It’s useful, and it has limits:

• It isn’t a lock. Anyone can read it, and RFC 9309 says outright that it isn’t access control. Private pages need a login.
• It doesn’t remove URLs. Google says it may still index a disallowed URL that is linked from elsewhere, just without a description. OpenAI says sites that opt out of ChatGPT search can still appear as navigational links. To keep a page out, let it be crawled and serve noindex, as we do on our API.
• It doesn’t bind user-triggered fetchers. OpenAI, Perplexity, Meta and Amazon all say their user-initiated agents may skip it.
• It doesn’t reach back in time. Anthropic describes blocking ClaudeBot as excluding your future content from training.
• It isn’t instant. OpenAI, Perplexity, Meta and Amazon each mention up to about 24 hours, and RFC 9309 lets crawlers cache the file for up to 24 hours.
• It doesn’t stop impostors. A scraper can call itself anything. Common Crawl warns about fake CCBots, which is why vendors publish IP lists.

Cloudflare and firewalls can overrule robots.txt

A firewall rule beats a text file. Cloudflare’s AI Crawler and Agent access policies (Security, Settings) can block search, agent and training crawlers at the edge, whatever your robots.txt says.

The defaults have moved this year. Cloudflare’s documentation, updated 2 November 2026, says new domains start with Search, Agents and Training all on Allow. Its July 2026 announcement had planned to block training and agent bots on pages with ads for new domains from 15 September 2026. Don’t trust either default: open your own zone and look.

Two more Cloudflare details catch people out. Its documentation says Training actions also apply to mixed-purpose crawlers that serve both search and training, and the July post named Googlebot, Applebot and Bingbot among them, so check what a Training block catches before you switch it on. And when you block a category, Cloudflare’s managed robots.txt prepends Disallow: / groups for Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent to your own file. The robots.txt crawlers receive can differ from the one in your CMS.

Mistakes we see that cost AI visibility

• Copy-pasting a “block all AI bots” list without reading it. The most dangerous mistake we see: a business makes itself invisible in AI answers to fix a training worry it never really had.
• A leftover Disallow: / under User-agent: * from a staging launch. It blocks every crawler without its own group, AI included.
• Blocking Googlebot to dodge AI Overviews. It takes the whole site out of Google Search.
• Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot when you only meant to opt out of training.
• A robots.txt that returns a server error. Under RFC 9309, crawlers must then treat the whole site as off limits.
• Blocking crawler IP ranges so the bot can’t even read your rules, or forgetting that every subdomain serves its own robots.txt.
• Keeping dead tokens such as Claude-Web and anthropic-ai. Our own file still has one.

How to test your robots.txt for AI crawlers

1. Open the live file at https://yourdomain.com/robots.txt, not the copy in your repository or CMS. Expect a 200 status and plain text, and repeat for each subdomain.
2. Test the tokens against a real page path with our free AI Crawler Tester. It reports Allowed or Blocked for 13 crawler tokens, including GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot and Google-Extended, and it flags a robots.txt that answers with a server error.
3. Check the syntax and any other user agent with the Robots.txt Tester and Validator, for example Googlebot or Applebot.
4. Look at Google’s view. Search Console’s robots.txt report shows which files Google fetched, when, and any lines it couldn’t parse, and it can request a recrawl in an emergency. It doesn’t test URLs: URL Inspection does that.
5. Check your CDN or firewall for AI crawler policies that block what the file allows.
6. Read your server logs and verify claimed bots against the vendors’ IP lists: openai.com/gptbot.json, openai.com/searchbot.json and openai.com/chatgpt-user.json for OpenAI, plus the Anthropic, Perplexity and Common Crawl lists above.
7. Give it a day. Most vendors mention up to 24 hours before a change shows up.

This is how the documented bots introduce themselves in a log line, as published by OpenAI, Perplexity and Common Crawl on 11 November 2026. OpenAI notes that its version numbers change, so search for the token, not the whole string:

GPTBot:          Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
OAI-SearchBot:   Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
ChatGPT-User:    Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
PerplexityBot:   Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Perplexity-User: Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)
CCBot:           CCBot/2.0 (https://commoncrawl.org/faq/)

A clean result means the rules let a crawler in. Whether that crawler then cites you is a different game, and that’s where GEO starts.

How this fits into GEO

Crawler access is the ticket in, not the result. After access, each platform still decides whether to index, retrieve, trust and cite a page. Our own snapshot above shows it: every door open, and still only a few mentions. The rest of the AI visibility stack does that work:

• Structured data: schema markup that helps AI systems cite you.
• A reading list for AI tools: robots.txt is the bouncer, llms.txt is the guide, and it can’t block anything. Draft one with our llms.txt Generator and check it with the llms.txt Checker.
• Content AI can quote: clear headings and direct answers, as in our guide to optimizing content for AI citations.
• Measurement: how to measure AI visibility.

To see how each platform picks its sources, read how AI search engines choose sources. For the big picture, start with our GEO guide or the GEO hub, and run a page through the free AI Search Visibility Checker for a crawler access and readiness check.

Rather hand it over? Our GEO services cover content, business facts, crawl access and repeated checks of what AI answers say about you. Or start with a Free website audit. 🤙