--- title: "X-Robots-Tag & Meta Robots Indexability Checker" description: "Inspect HTTP X-Robots-Tag headers, meta robots and Googlebot directives, then summarize whether a public URL exposes noindex or nofollow signals." canonical_url: "https://webaloha.co/tools/x-robots-tag-checker/" markdown_url: "https://webaloha.co/tools/x-robots-tag-checker.md" date_modified: "2026-07-12T00:00:00.000Z" --- # X-Robots-Tag & Noindex Checker Tool Online Last updated: Jul 12, 2026 Check whether a URL is indexable by reading both the HTTP X-Robots-Tag header and HTML robots meta tags. Built for technical SEO, GEO audits, and launch QA. ## How the X-Robots-Tag Checker Works This checker fetches the URL and inspects the two places search engines use for index directives. 1. **HTTP header check**, the API reads the X-Robots-Tag response header, including rules that affect non-HTML files. 2. **HTML robots check**, the tool extracts meta robots and googlebot directives from the page head. 3. **Directive parsing**, values such as noindex, nofollow, none, noarchive, nosnippet, and max-snippet are normalized. 4. **Effective verdict**, the result combines header and meta signals to show whether the URL appears indexable. ## Why Robots Directives Matter A single noindex directive can remove a revenue page from search, even if the page content, schema, and backlinks are strong. - **Launch protection**, staging noindex rules sometimes ship to production by accident. - **Header visibility**, X-Robots-Tag is invisible in page source, so many basic SEO tools miss it. - **AI search readiness**, GEO visibility depends on pages being crawlable, indexable, and machine-readable. - **Cleaner audits**, this tool complements the [Meta Tag Checker](https://webaloha.co/tools/meta-tag-checker/) by focusing only on index directives and HTTP headers. ## Robots Directive Reference | Directive | Meaning | SEO impact | | --- | --- | --- | | noindex | Do not show this URL in search results. | Blocks organic visibility. | | nofollow | Do not follow links on the page. | Can reduce discovery and link signal flow. | | none | Equivalent to noindex, nofollow. | Strong index and link block. | | noarchive | Do not show a cached copy. | Usually safe, but limits cache display. | | nosnippet | Do not show text snippets or previews. | Can lower click-through rate. | | max-snippet | Limit snippet text length. | Controls SERP display and AI extraction depth. | After checking directives, confirm real search visibility with the [Google Index Checker](https://webaloha.co/tools/google-index-checker/) and review crawl issues in our [technical SEO audit guide](https://webaloha.co/technical-seo-audit-guide/). ## X-Robots-Tag Directives Reference These are the indexing directives you can send in an X-Robots-Tag HTTP response header. Because the header is delivered with the HTTP response, it works on any file type, including PDFs, images, and other non-HTML resources where a meta robots tag is not possible. Each value below is read by compliant search engines such as Google. | Directive | Meaning | Note | | --- | --- | --- | | **noindex** | Do not show this URL in search results. | The main directive for removing a page or file from the index. | | **nofollow** | Do not follow links on this page. | Affects link discovery; does not by itself remove the page from the index. | | **none** | Equivalent to noindex, nofollow. | Shorthand that applies both blocks at once. | | **noarchive** | Do not store or show a cached copy. | Page can still be indexed and ranked; only the cached snapshot is hidden. | | **nosnippet** | Do not show a text or video snippet/preview. | Can reduce click-through because users see less context. | | **max-snippet:\[n\]** | Limit the snippet to n characters. | Use 0 for no snippet, -1 for no limit (e.g. max-snippet:160). | | **max-image-preview:\[setting\]** | Limit image preview size in results. | Accepts none, standard, or large. | | **noimageindex** | Do not index images on this page. | The page can still rank; its images will not appear in image search. | | **unavailable\_after:\[date\]** | Stop showing the URL after a given date/time. | Useful for expiring offers or time-limited content. | Reading robots headers correctly is one piece of a clean crawl setup. If indexability problems keep recurring, our [SEO services](https://webaloha.co/seo-services/) handle technical SEO end to end, from header rules to canonicals and crawl budget. Next steps ## X-Robots-Tag Checker related tools and articles Continue with the closest follow-up checks and guides based on this tool's topic, crawl intent, and optimization workflow. [![Google Index and Cache Checker Tool Online](https://webaloha.co/_astro/tool-index-checker.Ag8cBnXY_l6FHb.webp?dpl=dpl_H1MtSCmk524EGpEs7Etp4HWKr3Ti) Google Index Checker ](https://webaloha.co/tools/google-index-checker/)[![Technical SEO Audit: What to Check and How to Fix It](https://webaloha.co/_astro/blog-technical-seo-audit.DBRDzgnk_KFMKo.webp?dpl=dpl_H1MtSCmk524EGpEs7Etp4HWKr3Ti) Technical SEO Audit Guide ](https://webaloha.co/technical-seo-audit-guide/)[![Meta Tag Checker Tool Online](https://webaloha.co/_astro/tool-meta-checker.DJR8z9Ql_Z11a27K.webp?dpl=dpl_H1MtSCmk524EGpEs7Etp4HWKr3Ti) Meta Tag Checker ](https://webaloha.co/tools/meta-tag-checker/)[![Robots.txt Tester and Validator Tool](https://webaloha.co/_astro/tool-robots-tester.87NxGNuO_14XHx9.webp?dpl=dpl_H1MtSCmk524EGpEs7Etp4HWKr3Ti) Robots.txt Tester & Validator ](https://webaloha.co/tools/robots-txt-tester-and-validator/)[![Canonical URL Checker Tool Online](https://webaloha.co/_astro/tool-canonical-checker.CWZ_KgQd_1YgruQ.webp?dpl=dpl_H1MtSCmk524EGpEs7Etp4HWKr3Ti) Canonical URL Checker ](https://webaloha.co/tools/canonical-url-checker/)[![Broken Image Checker Tool Online](https://webaloha.co/_astro/tool-broken-images.p_a4yqXX_Z1y4sjS.webp?dpl=dpl_H1MtSCmk524EGpEs7Etp4HWKr3Ti) Broken Image Checker ](https://webaloha.co/tools/broken-image-checker/) ## X-Robots-Tag Checker: FAQ What directives does the checker inspect? It reads the X-Robots-Tag response header plus robots and googlebot meta tags in an HTML response. Values are split on commas, normalized to lowercase, and duplicates are removed; none is also treated as noindex and nofollow. How is the Indexable result calculated? The combined directive list is labeled non-indexable when it contains noindex or none. This is a narrow directive check and does not account for the HTTP status, robots.txt, canonicalization, authentication, rendering, or search-engine-specific indexing decisions. Does the result apply equally to every search engine? Not necessarily. The tool combines generic robots, googlebot meta, and the header into one summary rather than calculating a verdict per crawler. A Google-specific directive may not apply to Bing, and other bot-specific meta tags are not inspected. Can it interpret user-agent-specific X-Robots-Tag headers? Only simple comma-separated directives are reliably summarized. Header forms such as googlebot: noindex remain scoped text and are not fully parsed into per-agent rules, so inspect the displayed raw header for advanced configurations. Why are meta robots tags missing for a page I can see? Meta tags are read only when the fetched response declares a text/html content type, and the parser expects quoted name and content attributes. Tags injected by JavaScript, unusual markup, consent variants, or bot-specific responses can be missed. What does nofollow mean in this result? It asks compliant crawlers not to follow or pass signals through links on the page; it does not prevent the URL itself from being indexed. Use noindex when the intent is to keep that page out of search results. What should I do after finding an unexpected noindex? Check the final URL, raw X-Robots-Tag header, page template, CDN or server rules, SEO plugin settings, and deployment environment. Remove the unintended directive, purge relevant caches, and request a fresh crawl after verifying the live response. Is the checked URL or response stored? The URL is sent to WebAloha's server for a live request with a 12-second timeout. This endpoint does not save the submitted URL, HTML, headers, directives, or result to a database or result cache. Free 48-Hour Website Audit Not sure what to fix first on your own website? We'll review it and tell you, in plain English. Free & non-obligatory. [Get My Free Audit 💪](https://webaloha.co/free-website-audit/) ## Need a Full Indexability Audit? We find crawl blocks, noindex mistakes, canonical conflicts, and technical SEO issues before they cost traffic. [Book an SEO Audit](https://webaloha.co/seo-services/)