AI Crawler Robots.txt Tester

Last updated: Oct 2, 2026

Check how robots.txt rules match named crawler tokens on one URL path. Separate search, training and user-requested retrieval; the result does not prove real access, indexing or citations.

OpenAI trainingChatGPT searchChatGPT user requestsPerplexity searchPerplexity user requestsClaude trainingClaude searchClaude user requests + more

Why AI Crawler Access Matters

Your robots.txt file expresses crawling preferences. It is not authentication or a firewall. Search, training and user-requested retrieval use different provider tokens and can follow different policies.

Blocking GPTBot expresses an OpenAI training preference; it does not automatically block ChatGPT search. OAI-SearchBot handles search discovery, while user-initiated ChatGPT-User requests may not follow robots.txt. Allowing a token does not confirm that a provider visited, indexed or cited the page.

Use this report to review the rules for the submitted path, then inspect verified crawler logs and provider guidance if you need evidence of actual requests. Firewalls, login requirements and rendering can change access independently of robots.txt.

AI Crawlers This Tool Checks

This reference separates the named tokens and provider policies. A rule match is a preference, not a test of the provider requesting the page.

Platform	Bot Name	Type	Result meaning	Robots behavior	Provider reference and review
OpenAI training	`GPTBot`	Training	Training preference; independent of ChatGPT search.	Honors robots.txt	OpenAI documentation 2 October 2026
ChatGPT search	`OAI-SearchBot`	Search	Search discovery; not a citation prediction.	Honors robots.txt	OpenAI documentation 2 October 2026
ChatGPT user requests	`ChatGPT-User`	User request	User-initiated fetch; rule match may not govern access.	Robots rules may not apply	OpenAI documentation 2 October 2026
Perplexity search	`PerplexityBot`	Search	Search discovery, not foundation-model training.	Honors robots.txt	Perplexity documentation 2 October 2026
Perplexity user requests	`Perplexity-User`	User request	User-initiated fetch; rule match may not govern access.	Generally ignores robots.txt	Perplexity documentation 2 October 2026
Claude training	`ClaudeBot`	Training	Preference for future training collection.	Honors robots.txt	Anthropic documentation 2 October 2026
Claude search	`Claude-SearchBot`	Search	Search discovery and result quality.	Honors robots.txt	Anthropic documentation 2 October 2026
Claude user requests	`Claude-User`	User request	Retrieval initiated by a Claude user.	Honors robots.txt	Anthropic documentation 2 October 2026
Google Gemini controls	`Google-Extended`	Training / grounding	Training and specified Gemini grounding uses; not Google Search.	Robots product token, no separate request agent	Google documentation 2 October 2026
Bing search	`Bingbot`	Search	Bing crawling; not a standalone Copilot citation test.	Honors robots.txt	Microsoft documentation 2 October 2026
Common Crawl	`CCBot`	Dataset	Public crawl dataset used by multiple downstream consumers.	Honors robots.txt	Common Crawl documentation 2 October 2026
Meta token	`Meta-ExternalAgent`	Provider policy	Rule matching only; current provider policy was not reverified.	Review current provider documentation	Meta documentation Not reverified
Amazon	`Amazonbot`	Product / training	Amazon products and services; may contribute to model training.	Honors robots.txt	Amazon documentation 2 October 2026

Training Crawlers vs Retrieval Crawlers

Choose each policy intentionally. You can allow a provider’s search token while restricting its training token. User-requested retrieval is a third category and may have different robots behavior.

This is a rules simulator, so even a user-request token that generally ignores robots.txt receives a rule-match result. Read the provider note before treating it as an access restriction. Protect private content with authentication.

Cohere is not shown as an active training crawler because its current crawler page lists none. Anthropic’s documented search and user tokens replace the old generic anthropic-ai retrieval label in this reference.

For a policy workflow, read the robots.txt and AI search guide.

Related Tools

Robots.txt Tester & Validator, View and validate your full robots.txt file
Robots.txt Generator, Create a properly formatted robots.txt with AI bot controls
AI Search Visibility Checker, Broader technical AI search readiness screen
llms.txt Checker, Check if a site has a valid llms.txt file
llms.txt Generator, Create your AI content index file
Schema Markup Validator, Check your structured data

Next steps

AI Crawler Tester related tools and articles

Continue with the closest follow-up checks and guides based on this tool's topic, crawl intent, and optimization workflow.

Robots.txt for AI Search: Crawler Controls Without the Hype

llms.txt: The Complete, Evidence-Aware Guide

Generative Engine Optimization (GEO): Complete 2026 Guide

AI Search Visibility Checker

Robots.txt Tester & Validator

llms.txt Checker

AI Crawler Tester: FAQ

What exactly does the AI Crawler Tester check?

It fetches the robots.txt file from the submitted host, parses user-agent groups, and evaluates the submitted page path for the crawler tokens listed in the result table. It reports Allowed or Blocked for each token based only on those robots.txt rules.

Does Allowed mean an AI product will crawl or cite the page?

No. Allowed means the tested robots.txt rules do not prohibit that crawler token on that path. It does not confirm that the company currently uses the token, has discovered the page, has indexed its content, or will mention or cite it in an answer.

Why can different paths on the same site get different results?

Robots.txt rules can allow or disallow specific path prefixes. The tool evaluates the path you entered, so a crawler may be allowed on the homepage but blocked under a directory such as /private/. Test the actual page path that matters.

What does the tool report when robots.txt is missing?

If robots.txt returns 404 or has no readable content, the tool reports all tested crawlers as allowed by default. That reflects the absence of a robots exclusion rule, not proof that every crawler can pass a firewall, authentication layer, CDN rule, or server-level block.

How are training, search, and user-triggered crawler tokens different?

Crawler operators publish different tokens for different purposes, including model improvement, search discovery, and user-requested retrieval. A rule for one token does not automatically apply to another token from the same company unless a broader matching robots group also covers it. Set policy per published token and purpose.

What data is sent when I test crawler access?

The entered public URL is used to derive the host, page path, and robots.txt URL. The tool sends the robots.txt URL through the Web Aloha proxy first and may try a direct browser fetch as a fallback. It asks for no account credentials or personal details.

How do I allow or block specific AI crawlers?

Add User-agent rules to your robots.txt file. To allow a bot: "User-agent: GPTBot\nDisallow:" (empty disallow = allow all). To block a bot: "User-agent: GPTBot\nDisallow: /" (disallow all). Publish the file at the site root, then retest the exact page path and crawler token.

What are the limitations of this robots.txt test?

It tests the listed crawler tokens against one robots.txt file and one path. It cannot confirm crawler identity, server-level enforcement, JavaScript access, indexing, training use, or citation behavior. Robots.txt is a voluntary protocol, so this result applies to crawlers that honor it.

Free Website Audit

Not sure what to fix first on your own website? We'll review it and tell you, in plain English, within 48 hours. Free and non-obligatory.

Get my free website audit 💪

Need Help With Your Website?

We help businesses build and optimize their online presence, let's talk.

Explore GEO Services 🤙