Your robots.txt file gives crawl instructions to bots that respect the protocol. It does not enforce access control or guarantee indexing. An accidental block can prevent a crawler from fetching important content, while a blocked URL may still appear in search.
Here is how to check robots.txt of a website: find the file, read each rule, then test the URLs you care about. The steps work for your own site and for any other public site. After them come valid and invalid examples and the most common mistakes that cause real damage.
If you’re specifically interested in how robots.txt affects AI search and GEO, read our companion article on why robots.txt matters for AI search and GEO.
Using WordPress? Our WordPress robots.txt editing guide shows how to identify the physical file, virtual output, SEO plugin or host rule that controls the response, then verify your published change. Use this guide for syntax and crawl-rule testing.
How to Find Your Robots.txt File
Every robots.txt file lives at the same location: the root of your domain.
To check yours: Add /robots.txt to the end of your domain name in your browser:
https://yourdomain.com/robots.txtWhat you’ll see:
- A text file with rules: Your site has a robots.txt. Read on to learn how to interpret it.
- A 404 error: Your site doesn’t have a robots.txt. Google normally treats this as no crawl restrictions from robots.txt. This can be fine; authentication and other access controls still apply.
- An HTML page instead of plain text: Something is wrong. Your server is returning a webpage instead of a text file, which can confuse crawlers.
Important: The file must be at the exact root of your domain. It won’t work at yourdomain.com/folder/robots.txt or on a subdomain like blog.yourdomain.com/robots.txt (subdomains need their own separate robots.txt files).
How to Read a Robots.txt File (Syntax Reference)
Read a robots.txt file one group at a time: a User-agent line names the crawler, and the Disallow and Allow lines under it say which paths that crawler should not or may fetch. Robots.txt uses only a handful of directives. Here’s every directive you need to know:
The 5 Core Directives
| Directive | What It Does | Example |
|---|---|---|
| User-agent | Specifies which crawler the following rules apply to | User-agent: Googlebot |
| Disallow | Blocks crawlers from accessing a path | Disallow: /admin/ |
| Allow | Overrides a Disallow for a specific path | Allow: /admin/login |
| Sitemap | Tells crawlers where to find your sitemap | Sitemap: https://example.com/sitemap.xml |
| Crawl-delay | Adds delay between requests (not supported by Google) | Crawl-delay: 10 |
Syntax Rules You Must Follow
Wildcards and Pattern Matching
Google and Bing support two wildcard characters in robots.txt paths, even though they’re not part of the original specification:
| Character | Meaning | Example | Matches |
|---|---|---|---|
| * | Matches any sequence of characters | Disallow: /*.pdf | All PDF files anywhere on the site |
| $ | Matches the end of the URL | Disallow: /*.php$ | URLs ending in .php (not /page.php?id=1) |
Valid vs. Invalid Robots.txt: Side-by-Side Examples
Understanding what makes a robots.txt valid or broken is easier with examples:
Example 1: Allow all crawlers, block admin
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Sitemap: https://example.com/sitemap.xmlUser-agent: * Disallow: /admin/
Allow /
sitemap: sitemap.xmlIssues: directive on same line as User-agent, missing colon after Allow, sitemap URL is relative (needs full https:// URL)
Example 2: Block specific bot from specific directory
User-agent: GPTBot
Disallow: /private/
Disallow: /customer-data/
User-agent: *
Allow: /User-agent: GPTBot
Disallow: private/
Disallow: customer-data
User-agent: *
Allow: /Issues: paths missing leading / (must be /private/ not private/). Second path missing trailing / so it only blocks the exact file “customer-data” not the directory.
Example 3: Block everything (the nuclear option)
User-agent: *
Disallow: /This asks compliant crawlers matching this group not to fetch any path. It does not guarantee removal from search or prevent access by people or noncompliant bots. Protect private staging sites with authentication.
User-agent: *
Allow: /
Disallow: /staging/
Disallow: /test/Allow crawling except for the listed directories. These paths remain publicly accessible unless protected separately.
How to Test Your Robots.txt: 3 Methods
Method 1: Robots.txt tester tool (quickest)
Use a robots.txt tester tool to analyze your file instantly:
- Enter your website URL
- The tool fetches your robots.txt and parses every directive
- See which user agents have rules, what’s allowed, what’s blocked
- Check specific URLs against specific bots to verify behavior
This is the fastest way to catch syntax errors, conflicting rules, and accidental blocks.
Method 2: Google Search Console (most authoritative)
Google Search Console has a built-in robots.txt report:
- Go to Google Search Console
- Select your property
- Navigate to Settings in the left sidebar
- Look for the robots.txt section
- Google shows the last crawled version, any warnings, and errors
What to look for:
- File and fetch status: Confirm which host and robots.txt Google retrieved
- Issues: Review reported fetch or parsing problems and the actual directives
- Last checked: Compare the fetched version with your current published file
The current report shows Google’s fetched file and crawl issues; it is not the retired arbitrary URL and user-agent tester. Use URL Inspection for Google’s view of an individual URL, and the robots.txt tester for rule simulation. Simulation does not prove a bot actually fetched or indexed a page. Report scope checked 2 October 2026.
Method 3: Manual review (for simple files)
For small robots.txt files, you can review manually:
- Go to
yourdomain.com/robots.txt - Read through each group (each User-agent: section)
- Verify that important pages aren’t accidentally blocked
- Check that a Sitemap line is present
- Look for the AI crawler rules we covered in why robots.txt matters for AI search
The 10 Most Common Robots.txt Mistakes
These are the mistakes we see most often when testing client websites. Some of them can cause serious SEO damage.
1. Blocking your entire site by accident
The mistake: Leaving Disallow: / under User-agent: * from a development or staging environment.
How it happens: You block crawlers during development, then push the staging robots.txt to production. Or a CMS plugin adds a blanket disallow that you don’t notice.
The fix: Always verify your robots.txt after deploying to production. Set up monitoring to alert you if the file changes.
2. No robots.txt file at all
The behavior: A 404 for robots.txt normally means Google has no crawl restrictions from that file. A missing file is not automatically a defect. You can submit a sitemap separately; add robots.txt when you need crawl rules or sitemap discovery there.
The next step: Decide whether crawl rules are needed. If they are, use our Robots.txt Generator and test the intended rules.
3. Missing leading slash on paths
The mistake: Writing Disallow: admin/ instead of Disallow: /admin/.
Why it matters: Paths in robots.txt must start with /. Without it, the rule may not be recognized or may behave unpredictably.
The fix: Always start paths with /.
4. Confusing Disallow with noindex
The mistake: Using Disallow when you actually want to prevent a page from appearing in search results.
Why it matters: Disallow tells crawlers not to crawl a page, but Google can still index and display the URL in search results if other sites link to it. It may appear without a content-derived snippet.
The fix: For pages you want removed from search results, use a noindex meta tag or X-Robots-Tag header. Google must be able to crawl the page to read noindex. Use authentication for private content.
5. Conflicting Allow and Disallow rules
The mistake: Having rules that contradict each other:
User-agent: *
Disallow: /blog/
Allow: /blog/How Google resolves it: When Allow and Disallow rules conflict, Google uses the most specific rule (the one with the longest path). If they’re the same length, Google defaults to Allow. But other crawlers may handle this differently.
The fix: Be explicit. Don’t rely on conflict resolution rules. Make your intent clear with non-overlapping paths.
6. Blocking CSS, JavaScript, or images
The mistake: Blocking crawlers from accessing CSS, JS, or image files that are needed to render your pages.
Why it matters: Google needs to render your pages to understand their content. If you block CSS or JavaScript, Google can’t see your page the way users do. This can harm your rankings and prevent proper indexing.
The fix: Don’t block /css/, /js/, /images/, or /assets/ directories unless you have a specific reason. Check Google’s URL Inspection tool to see if rendering issues exist.
7. Wrong file location
The mistake: Placing robots.txt in a subdirectory like /pages/robots.txt instead of the root.
Why it matters: Crawlers only check the root domain for robots.txt. A file anywhere else is ignored.
The fix: robots.txt must be at https://yourdomain.com/robots.txt. No exceptions.
8. Protocol mismatch (http vs https)
The mistake: Having robots.txt only on http:// when your site runs on https:// (or vice versa).
Why it matters: Robots.txt rules only apply to the protocol where the file is hosted. If your site is HTTPS but your robots.txt is only accessible via HTTP, HTTPS crawling is unaffected.
The fix: Ensure robots.txt is accessible at the same protocol your site uses. If you redirect HTTP to HTTPS, make sure the robots.txt is available at the HTTPS URL.
9. Relative sitemap URL
The mistake: Using Sitemap: /sitemap.xml instead of the full URL.
Why it matters: The Sitemap directive requires an absolute URL including the protocol and domain.
The fix: Always use the full URL: Sitemap: https://yourdomain.com/sitemap.xml
10. Not updating for AI crawlers
The issue to review: Whether the applicable rules match your policy for each crawler’s purpose.
Why it matters: Without explicit rules for AI bots, they fall under your User-agent: * rule, which may not reflect your actual intent. You’re either letting all AI bots in by default or blocking them all by accident.
The next step: Separate search access from training permissions. Add bot-specific rules only where your intent differs from the applicable general rule. Our AI crawler guide links the providers’ current purpose and control documentation.
Robots.txt vs. Meta Robots vs. X-Robots-Tag
These three mechanisms all control crawler behavior, but they do different things:
| Method | Scope | Controls | Use When |
|---|---|---|---|
| robots.txt | Entire site or directories | Crawling (what bots can access) | Managing crawl budget, blocking directories, controlling AI bot access |
| Meta robots tag | Individual pages | Indexing (noindex, nofollow) | Preventing specific pages from appearing in search results |
| X-Robots-Tag | Individual URLs (HTTP header) | Indexing (noindex, nofollow) | Non-HTML files (PDFs, images) or when you can’t edit page HTML |
Key insight: robots.txt controls compliant crawling; meta robots and X-Robots-Tag can control indexing when crawlers can read them. Authentication protects private content. Google’s robots introduction, checked 2 October 2026, explains these limits. llms.txt is a separate community proposal, not an access control or established requirement for AI visibility.
Robots.txt Checklist: Before Going Live
Use this checklist every time you create or update a robots.txt file:
Next Steps
Your robots.txt is one piece of your site’s technical SEO foundation. Here’s how everything connects:
- Test your current file with our Robots.txt Tester to find issues instantly
- Generate a new file with our Robots.txt Generator that includes AI crawler toggles
- Understand the AI angle in our guide on why robots.txt matters for AI search and GEO
- Check your full SEO foundation by validating your sitemap, meta tags, and schema markup
- Audit AI search readiness with our AI Search Visibility Checker that includes robots.txt analysis
- Get professional help through our SEO services or GEO services
Test Your Robots.txt Now
Find syntax errors, conflicting rules, and accidental blocks. Free, instant results.
Frequently asked questions
How do I find my website's robots.txt file?
Add /robots.txt to the end of your domain name. For example, if your website is example.com, go to example.com/robots.txt in your browser. If a robots.txt file exists, you'll see its contents as plain text. If you get a 404 error, your site doesn't have one. The file must be in the root directory of your domain to work.
What happens if my website has no robots.txt file?
A 404 for robots.txt normally means Google has no crawl restrictions from that file. This can be fine for a public site. Add the file when you need crawl rules or sitemap discovery there; authentication and indexing controls remain separate.
What is the difference between Disallow and noindex?
Disallow in robots.txt tells crawlers not to crawl a page, but it does not prevent the page from appearing in search results. If other pages link to a disallowed URL, Google can still index it and show it in search results without crawling it. The noindex meta tag tells Google not to include the page in search results at all. For noindex to work, the crawler must be allowed to fetch and read it. For pages you just want to save crawl budget on, use Disallow.
Is robots.txt case-sensitive?
Yes. Path rules in robots.txt are case-sensitive. Disallow: /Photos/ blocks /Photos/ but not /photos/. Directive names (User-agent, Disallow, Allow) are case-insensitive, but the URL paths after them are case-sensitive. This is a common source of mistakes.
How do I test if my robots.txt is working correctly?
Three ways: Use Google Search Console's robots.txt report to see what Google found and any errors. Use a robots.txt tester tool to paste your file and check specific URLs against specific user agents. Or view your file directly at yourdomain.com/robots.txt and manually verify the rules. Always test after making changes.
What does 'User-agent: *' mean in robots.txt?
The asterisk (*) is a wildcard that matches all crawlers. Rules under 'User-agent: *' apply to every bot that doesn't have its own specific section. If you also have a section for a specific bot like Googlebot, that specific section takes priority for Googlebot, and the wildcard rules apply to all other bots.
Can I use robots.txt to remove a page from Google?
No. Disallow in robots.txt prevents Google from crawling a page, but Google can still index and display the URL in search results if other pages link to it. To remove a page from Google, use the noindex meta tag or the X-Robots-Tag HTTP header. Allow crawling so Google can read noindex. The Search Console removals tool can request temporary removal.
How do I fix 'Blocked by robots.txt' in Search Console?
Find the Disallow rule that matches the URL, then remove or narrow it if the page should be crawled, and confirm with URL Inspection's live test. If you actually want the page out of Google, allow crawling and use noindex instead, because a blocked URL can still be indexed without its content.
What replaced Google's robots.txt Tester?
Google retired the robots.txt Tester in late 2023 and replaced it with the robots.txt report in Search Console, which shows the robots.txt files Google found, when it last checked them and any parsing issues. URL Inspection shows whether Google can crawl one URL; to test new rules against any URL before publishing, use a third-party robots.txt tester.
How often should I review my robots.txt file?
Review your robots.txt after any site redesign, URL structure change, CMS migration, or plugin update. Also review it quarterly to ensure no accidental changes have been made. With the rise of AI crawlers, you should also review it whenever new AI bots emerge to decide whether to allow or block them.
