X-Robots-Tag & Noindex Checker Tool Online

Last updated:

Read HTTP X-Robots-Tag and HTML robots directives for one response. No noindex detected is a narrow result, not proof of Google indexing eligibility.

Enter a URL to check robots index directives:

How the X-Robots-Tag Checker Works

This checker fetches the URL and inspects the two places search engines use for index directives.

  1. HTTP header check, the API reads the X-Robots-Tag response header, including rules that affect non-HTML files.
  2. HTML robots check, the tool extracts meta robots and googlebot directives from the page head.
  3. Directive parsing, values such as noindex, nofollow, none, noarchive, nosnippet, and max-snippet are normalized.
  4. Effective verdict, the result combines header and meta signals to show whether a noindex or none directive was detected.

Why Robots Directives Matter

A single noindex directive can remove a revenue page from search, even if the page content, schema, and backlinks are strong.

  • Launch protection, staging noindex rules sometimes ship to production by accident.
  • Header visibility, X-Robots-Tag is invisible in page source, so many basic SEO tools miss it.
  • AI search readiness, GEO visibility depends on pages being crawlable, indexable, and machine-readable.
  • Cleaner audits, this tool complements the Meta Tag Checker by focusing only on index directives and HTTP headers.

X-Robots-Tag Directives Reference

These are the indexing directives you can send in an X-Robots-Tag HTTP response header. Because the header is delivered with the HTTP response, it works on any file type, including PDFs, images, and other non-HTML resources where a meta robots tag is not possible. Each value below is read by compliant search engines such as Google.

DirectiveMeaningNote
noindexDo not show this URL in search results.The main directive for removing a page or file from the index.
nofollowDo not follow links on this page.Affects link discovery; does not by itself remove the page from the index.
noneEquivalent to noindex, nofollow.Shorthand that applies both blocks at once.
noarchiveHistorical request to suppress cached copies; ignored by Google.It has no current Google cached-copy effect. Other engines may differ.
nosnippetDo not show a text or video snippet/preview.Can reduce click-through because users see less context.
max-snippet:[n]Limit the snippet to n characters.Use 0 for no snippet, -1 for no limit (e.g. max-snippet:160).
max-image-preview:[setting]Limit image preview size in results.Accepts none, standard, or large.
noimageindexDo not index images on this page.Images may still be discovered and indexed from other pages. This does not noindex the page itself.
unavailable_after:[date]Stop showing the URL after a given date/time.Useful for expiring offers or time-limited content.

Reading robots headers correctly is one piece of a clean crawl setup. If indexability problems keep recurring, our SEO services handle technical SEO end to end, from header rules to canonicals and crawl budget.

Next steps

X-Robots-Tag Checker related tools and articles

Continue with the closest follow-up checks and guides based on this tool's topic, crawl intent, and optimization workflow.

Google Index Checker

Technical SEO Audit Guide

Meta Tag Checker

How to Check and Test Your Robots.txt File: The Complete Guide

Google Search Console Guide for Business Owners

Robots.txt for AI Search: Crawler Controls Without the Hype

X-Robots-Tag Checker: FAQ

What directives does the checker inspect?
It reads the X-Robots-Tag response header plus robots and googlebot meta tags in an HTML response. Values are split on commas, normalized to lowercase, and duplicates are removed; none is also treated as noindex and nofollow.
How is the noindex result calculated?
The combined directive list is labeled Noindex detected when it contains noindex or none. This is a narrow directive check and does not account for the HTTP status, robots.txt, canonicalization, authentication, rendering, or search-engine-specific indexing decisions.
Does the result apply equally to every search engine?
Not necessarily. The tool combines generic robots, googlebot meta, and the header into one summary rather than calculating a verdict per crawler. A Google-specific directive may not apply to Bing, and other bot-specific meta tags are not inspected.
Can it interpret user-agent-specific X-Robots-Tag headers?
Only simple comma-separated directives are reliably summarized. Header forms such as googlebot: noindex remain scoped text and are not fully parsed into per-agent rules, so inspect the displayed raw header for advanced configurations.
Why are meta robots tags missing for a page I can see?
Meta tags are read only when the fetched response declares a text/html content type, and the parser expects quoted name and content attributes. Tags injected by JavaScript, unusual markup, consent variants, or bot-specific responses can be missed.
What does nofollow mean in this result?
It asks compliant crawlers not to follow or pass signals through links on the page; it does not prevent the URL itself from being indexed. Use noindex when the intent is to keep that page out of search results.
What should I do after finding an unexpected noindex?
Check the final URL, raw X-Robots-Tag header, page template, CDN or server rules, SEO plugin settings, and deployment environment. Remove the unintended directive, purge relevant caches, and request a fresh crawl after verifying the live response.
Is the checked URL or response stored?
The URL is sent to WebAloha's server for a live request with a 12-second timeout. This endpoint does not save the submitted URL, HTML, headers, directives, or result to a database or result cache.

Free Website Audit

Not sure what to fix first on your own website? We'll review it and tell you, in plain English, within 48 hours. Free and non-obligatory.

Get my free website audit 💪

Need a Full Indexability Audit?

We find crawl blocks, noindex mistakes, canonical conflicts, and technical SEO issues before they cost traffic.

Book an SEO Audit