---
title: "Edit WordPress robots.txt and Verify the File Crawlers Receive"
description: "Find whether WordPress, a physical file, an SEO plugin or your host controls robots.txt, then verify the published rules with a practical before-and-after test."
canonical_url: "https://webaloha.co/wordpress-robots-txt/"
markdown_url: "https://webaloha.co/wordpress-robots-txt.md"
date_published: "2026-10-28T00:00:00.000Z"
date_modified: "2026-10-28T00:00:00.000Z"
author: "https://webaloha.co/authors/lucky-oleg/"
---
# Edit WordPress robots.txt and Verify the File Crawlers Receive

Author: [Lucky Oleg](https://webaloha.co/authors/lucky-oleg/) | Published Oct 28, 2026

**Edit the source that actually serves your WordPress site’s `/robots.txt`, then fetch that exact public URL to verify the result.** It may be a physical file, WordPress’s virtual output, an SEO plugin or a host/CDN rule. Saving text in a plugin screen does not establish that crawlers receive it.

This guide is about that WordPress implementation problem. For directive syntax, wildcard rules and broader crawler questions, use our [robots.txt checking guide](https://webaloha.co/how-to-check-robots-txt/). You can paste the examples below into the [Robots.txt Tester and Validator](https://webaloha.co/tools/robots-txt-tester-and-validator/) before publishing a change.

## 1\. Find the robots.txt URL for the public site

If the public site is `https://example.com`, inspect:

```
https://example.com/robots.txt
```

The URL is at the host’s root, even if WordPress files are installed under `/wordpress/`. A file at `/wordpress/robots.txt` is not the root robots.txt for that host. Different subdomains and schemes have their own scope; a rule for one origin does not automatically apply to another. [Google documents the location and scope](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec).

Use the production scheme and hostname, and record any redirect. A browser can show an HTML error page at an address ending in `.txt`, so inspect the response as well as the URL.

## 2\. Determine what controls the response

Start with **Tools → Site Health → Info → Server** and the hosting file manager. WordPress’s [Site Health detection](https://developer.wordpress.org/reference/classes/wp_debug_data/get_wp_server/) checks for a static root file and reports whether WordPress can serve the dynamic version. That is a useful origin check; a reverse proxy or custom routing rule can still change what the public receives.

| Possible source | How to investigate | Where to make the change |
| --- | --- | --- |
| CDN or edge rule | Compare the public response with the host’s approved origin diagnostic | The edge rule or managed file, then the relevant cache |
| Host or web-server route | Ask whether `/robots.txt` is served separately from WordPress | The host-managed rule or file |
| Physical file in the site’s document root | Inspect the actual root file and compare its body with the public response | That file or the deployment system that owns it |
| WordPress virtual response | Confirm no physical file is served and the request reaches WordPress | The active plugin or site-specific implementation that filters the output |
| SEO plugin editor | Check whether it writes a physical file or changes virtual output | The relevant plugin feature, after identifying its mechanism |

With standard WordPress rewrite rules, an existing physical file is normally served directly instead of going through the virtual route. That makes a physical `robots.txt` a common reason a WordPress filter appears to do nothing. Custom server configurations can differ, so verify rather than assuming a universal priority order.

Several plugins can filter virtual output, and a plugin’s file editor can instead create a physical file. Do not add a second SEO plugin just to obtain another editor. Record one owner for the rules and remove conflicting instructions only after comparing their purpose.

### What WordPress core generates

The current [`do_robots()` implementation](https://developer.wordpress.org/reference/functions/do_robots/) creates a wildcard crawler group, blocks the path returned by `admin_url()`, allows its `admin-ajax.php` path, and passes the result through `robots_txt` filters. For a typical root installation, those core path rules are:

```
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
```

Other WordPress features can add output. The [core sitemap integration](https://developer.wordpress.org/reference/classes/wp_sitemaps/add_robots/) can append its sitemap index for a public site when that integration is active. An SEO plugin can provide a different sitemap. Verify the live sitemap URL rather than copying `/wp-sitemap.xml` or `/sitemap_index.xml` without checking.

The official reference page also contains user-contributed examples. Those notes are separate from core behavior; a sample that blocks images, fonts or scripts is not a default recommendation for your site.

## 3\. Choose a bounded way to edit it

Before changing anything, save the current body, HTTP status, headers and time. Write down the intended effect on actual URLs, such as “allow the service page and stylesheet; keep the admin area disallowed.”

### If a physical file is authoritative

Use the hosting file manager or SFTP to edit the existing root `robots.txt` as plain UTF-8 text. Preserve the working version and its deployment source. A manual edit can be overwritten if the file is generated during releases, so update the owning configuration as well.

Do not create a new root file merely because a virtual file cannot be found in the file manager. That virtual file can be working correctly, and a new physical file can replace plugin-provided output.

### If a plugin owns virtual output

Use its documented robots feature, preview the text where possible and save a single change. Plugin menus and storage mechanisms differ by version. Check the plugin’s current documentation and re-fetch the public file after publishing.

For code-managed sites, a developer can use the [`robots_txt` filter](https://developer.wordpress.org/reference/hooks/robots_txt/) in a site-specific plugin. The filter only affects requests that reach WordPress’s virtual route. It cannot change a file already served by the web server or an edge response. Use staging for PHP edits and retain a way to undo the change without the WordPress dashboard.

### If the host or CDN owns it

Ask the provider to change the effective rule or managed file. Send the requested body and expected URL decisions. Purge the relevant cache after the origin change is confirmed, then inspect the public response again. Do not treat a cache purge as proof that the source is correct.

Our [Robots.txt Generator](https://webaloha.co/tools/robots-txt-generator/) can help prepare rules. Its output still needs review against the site’s real public pages, assets, sitemap and existing crawler policies.

## 4\. Verify the file served over HTTP

A GET request checks the response body that the public endpoint serves. For a site you administer, a command-line capture can be:

```
curl --silent --show-error --location --max-redirs 5 --max-time 30 \
  --dump-header robots-headers.txt --output robots-body.txt \
  https://example.com/robots.txt
```

Replace the example domain, inspect every response header block and open `robots-body.txt`. Keep the final scheme and host in scope. Check for the expected plain-text rules, an appropriate successful status, redirects and a cache header that explains stale output. An HTML login page, security challenge or 404 document is a different response from the intended file.

For a browser-based check, use our [tester](https://webaloha.co/tools/robots-txt-tester-and-validator/) to fetch the public file. Compare its returned body with the text you edited. Run it again after publishing rather than relying on a preview.

A public fetch establishes what that request received. It does not prove that Google has already refreshed its cached copy, that every edge location is identical, or that the page is indexed.

## A checked before-and-after example

Here is a deliberately broken configuration for a public site:

```
User-agent: *
Disallow: /
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/wp-sitemap.xml
```

For this fixture, the intended change is to allow public pages and rendering assets while keeping the admin path disallowed:

```
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/wp-sitemap.xml
```

These are examples, not a universal production file. Replace the sitemap with your verified URL and retain any deliberate site-specific policies. A store’s filters, search URLs or other special cases need their own review.

On **28 October 2026**, we ran both exact bodies through the parser and path-simulation functions used by our tester. We checked the following paths as both `Googlebot` and `Bingbot`; both selected the wildcard group and produced the same decisions:

| Tested path | Before | After |
| --- | --- | --- |
| `/` | Disallowed | Allowed |
| `/services/` | Disallowed | Allowed |
| `/wp-admin/` | Disallowed | Disallowed |
| `/wp-admin/admin-ajax.php?action=example` | Allowed | Allowed |
| `/wp-content/themes/example/style.css` | Disallowed | Allowed |

The AJAX path remains allowed in the broken example because its more specific `Allow` beats the broad `Disallow`. This is why reading only one line can miss the actual decision.

This was a **local rules diagnostic**, not an observation of Google or Bing crawling a WordPress installation. It checks these path patterns and group selection; it does not test CDN behavior, crawler refresh times, indexing or every robots extension. Google’s [robots specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec) is the primary reference for Google-specific behavior.

Download the [before fixture](https://webaloha.co/downloads/wordpress-troubleshooting/robots-before.txt), [after fixture](https://webaloha.co/downloads/wordpress-troubleshooting/robots-after.txt) and [verification checklist](https://webaloha.co/downloads/wordpress-troubleshooting/robots-verification-checklist.txt) to repeat the comparison with your site’s important paths.

## Search visibility is a separate WordPress setting

**Settings → Reading → Discourage search engines from indexing this site** does not mean “make this website private.” WordPress’s [`wp_robots_noindex()`](https://developer.wordpress.org/reference/functions/wp_robots_noindex/) adds page-level indexing instructions when the site is not public. Core removed the automatic site-wide `Disallow: /` from `do_robots()` in WordPress 5.3. Plugins or host rules can still add their own instructions.

If a public site is accidentally discouraged, uncheck the setting after verifying that it is intended to be public. Then inspect an actual page’s rendered robots meta tag and HTTP headers, alongside `/robots.txt`. Our [X-Robots-Tag Checker](https://webaloha.co/tools/x-robots-tag-checker/) helps inspect header directives; plugin or server-level rules may still apply after the Reading setting changes.

A robots crawl block cannot reliably remove an indexed URL. It can also stop Google from reading a page’s `noindex` instruction. Follow [Google’s noindex guidance](https://developers.google.com/search/docs/crawling-indexing/block-indexing) for public pages that should leave search, and require authentication for private content. Robots rules are requests to compliant crawlers, not access controls.

## Subdirectories and multisite need an origin-level check

For WordPress installed in a subdirectory, inspect the real admin URLs before copying the root-installation example. Core derives the paths from `admin_url()`, so `/wordpress/wp-admin/` may be appropriate where `/wp-admin/` is not.

For multisite under one host, plan root robots rules across the network’s public paths. A subsite administrator’s plugin settings do not by themselves establish which site or routing layer generates the root response. For mapped domains or subdomains, inspect `/robots.txt` on each relevant public origin. Avoid a shared root file that accidentally blocks another site’s pages.

Keep a representative public page, required stylesheet or script, intended restricted path and sitemap in the acceptance check for each origin. After the edit, fetch the final body, repeat the path tests and inspect an important live page in Search Console when needed. Passing a local rules test does not guarantee indexing or ranking.

## Get help fixing the WordPress configuration

Send the public robots.txt URL, the body you intended to publish, the source you edited and one URL with an unexpected decision. Our [Website Fix Sprint](https://webaloha.co/website-fix-sprint/) can cover an agreed configuration priority after assessment. We confirm the scope, price and timeline before starting.

If the underlying maintenance also requires a runtime update, our [WordPress PHP update service](https://webaloha.co/wordpress-php-update-service/) explains its separate scope. Ongoing [WordPress care](https://webaloha.co/wordpress-website-maintenance-services/) is available for continued maintenance.

**[Get help fixing my WordPress robots.txt](https://webaloha.co/contact/?service=Website%20Maintenance&request=wordpress-robots-txt)**

## Frequently asked questions

Where is robots.txt in WordPress?

Crawlers request /robots.txt at the root of the relevant scheme and host. WordPress can generate a virtual response when no physical file is served. A host, web-server rule, CDN or plugin can affect the response, so seeing /robots.txt in a browser does not prove there is a file in the WordPress directory.

Why do my WordPress robots.txt edits not appear?

Check that you edited the source actually serving /robots.txt. A physical root file can bypass WordPress's virtual output, an SEO plugin may use a filter or write a real file, and a host or CDN may serve another response or an older cached copy. Verify the exact public URL with a GET request after publishing.

Does Discourage search engines add Disallow slash to WordPress robots.txt?

Current WordPress core does not use that setting to add a site-wide Disallow rule in its default robots.txt output. That behavior was removed in WordPress 5.3 in favor of page-level robots controls. Check the rendered robots meta tag, response headers and any plugin or host overrides.

Does blocking a WordPress URL in robots.txt remove it from Google?

No. robots.txt controls compliant crawling. Google can still know a blocked URL from links, and a crawl block can prevent it from seeing noindex on the page. Use appropriate indexing controls for public pages and authentication for private content.

Recommended tools

## Recommended tools for this guide

Use these free tools to apply the ideas from this guide to your own website.

[![Robots.txt Generator Tool Online](https://webaloha.co/_astro/tool-robots-generator.EvCPtaQL_r12dE.webp?dpl=dpl_361ezMzWCgtUtFxFQVMru9SddmJo)

Robots.txt Generator

](https://webaloha.co/tools/robots-txt-generator/)[![WordPress PHP Upgrade Readiness Checker Tool](https://webaloha.co/_astro/tool-wordpress-php-readiness.CWOsWIns_ZNHK1B.webp?dpl=dpl_361ezMzWCgtUtFxFQVMru9SddmJo)

WordPress PHP Upgrade Checker

](https://webaloha.co/tools/wordpress-php-upgrade-readiness-checker/)[![WordPress Version Exposure Checker Tool Online](https://webaloha.co/_astro/tool-wp-version.DONOq27n_23vbEp.webp?dpl=dpl_361ezMzWCgtUtFxFQVMru9SddmJo)

WordPress Version Checker

](https://webaloha.co/tools/wordpress-version-checker/)

Useful info? Spread the Aloha:

[![Lucky Oleg](https://webaloha.co/_astro/lucky-oleg.EPOIIvwX_ZX2S03.webp?dpl=dpl_361ezMzWCgtUtFxFQVMru9SddmJo)](https://webaloha.co/authors/lucky-oleg/)

[Lucky Oleg](https://webaloha.co/authors/lucky-oleg/)[](https://www.linkedin.com/in/luckyoleg/)

Lucky Oleg is the founder of Web Aloha, a web design & SEO agency helping businesses ride the digital wave. With years of experience in WordPress, technical SEO, and web performance, he writes about what actually works in the real world.

#### Related guides from Web Aloha

[![](https://webaloha.co/_astro/blog-robots-txt-ai-geo.D_4d0VJ2_Z1cXCWI.webp?dpl=dpl_361ezMzWCgtUtFxFQVMru9SddmJo)

Robots.txt for AI Search: Crawler Controls Without the Hype

](https://webaloha.co/robots-txt-ai-search-geo/)