Edit WordPress robots.txt and Verify the File Crawlers Receive

Edit the source that actually serves your WordPress site’s /robots.txt, then fetch that exact public URL to verify the result. It may be a physical file, WordPress’s virtual output, an SEO plugin or a host/CDN rule. Saving text in a plugin screen does not establish that crawlers receive it.

This guide is about that WordPress implementation problem. For directive syntax, wildcard rules and broader crawler questions, use our robots.txt checking guide. You can paste the examples below into the Robots.txt Tester and Validator before publishing a change.

1. Find the robots.txt URL for the public site

If the public site is https://example.com, inspect:

https://example.com/robots.txt

The URL is at the host’s root, even if WordPress files are installed under /wordpress/. A file at /wordpress/robots.txt is not the root robots.txt for that host. Different subdomains and schemes have their own scope; a rule for one origin does not automatically apply to another. Google documents the location and scope.

Use the production scheme and hostname, and record any redirect. A browser can show an HTML error page at an address ending in .txt, so inspect the response as well as the URL.

2. Determine what controls the response

Start with Tools → Site Health → Info → Server and the hosting file manager. WordPress’s Site Health detection checks for a static root file and reports whether WordPress can serve the dynamic version. That is a useful origin check; a reverse proxy or custom routing rule can still change what the public receives.

With standard WordPress rewrite rules, an existing physical file is normally served directly instead of going through the virtual route. That makes a physical robots.txt a common reason a WordPress filter appears to do nothing. Custom server configurations can differ, so verify rather than assuming a universal priority order.

Several plugins can filter virtual output, and a plugin’s file editor can instead create a physical file. Do not add a second SEO plugin just to obtain another editor. Record one owner for the rules and remove conflicting instructions only after comparing their purpose.

What WordPress core generates

The current do_robots() implementation creates a wildcard crawler group, blocks the path returned by admin_url(), allows its admin-ajax.php path, and passes the result through robots_txt filters. For a typical root installation, those core path rules are:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Other WordPress features can add output. The core sitemap integration can append its sitemap index for a public site when that integration is active. An SEO plugin can provide a different sitemap. Verify the live sitemap URL rather than copying /wp-sitemap.xml or /sitemap_index.xml without checking.

The official reference page also contains user-contributed examples. Those notes are separate from core behavior; a sample that blocks images, fonts or scripts is not a default recommendation for your site.

3. Choose a bounded way to edit it

Before changing anything, save the current body, HTTP status, headers and time. Write down the intended effect on actual URLs, such as “allow the service page and stylesheet; keep the admin area disallowed.”

If a physical file is authoritative

Use the hosting file manager or SFTP to edit the existing root robots.txt as plain UTF-8 text. Preserve the working version and its deployment source. A manual edit can be overwritten if the file is generated during releases, so update the owning configuration as well.

Do not create a new root file merely because a virtual file cannot be found in the file manager. That virtual file can be working correctly, and a new physical file can replace plugin-provided output.

If a plugin owns virtual output

Use its documented robots feature, preview the text where possible and save a single change. Plugin menus and storage mechanisms differ by version. Check the plugin’s current documentation and re-fetch the public file after publishing.

For code-managed sites, a developer can use the robots_txt filter in a site-specific plugin. The filter only affects requests that reach WordPress’s virtual route. It cannot change a file already served by the web server or an edge response. Use staging for PHP edits and retain a way to undo the change without the WordPress dashboard.

If the host or CDN owns it

Ask the provider to change the effective rule or managed file. Send the requested body and expected URL decisions. Purge the relevant cache after the origin change is confirmed, then inspect the public response again. Do not treat a cache purge as proof that the source is correct.

Our Robots.txt Generator can help prepare rules. Its output still needs review against the site’s real public pages, assets, sitemap and existing crawler policies.

4. Verify the file served over HTTP

A GET request checks the response body that the public endpoint serves. For a site you administer, a command-line capture can be:

curl --silent --show-error --location --max-redirs 5 --max-time 30 \
  --dump-header robots-headers.txt --output robots-body.txt \
  https://example.com/robots.txt

Replace the example domain, inspect every response header block and open robots-body.txt. Keep the final scheme and host in scope. Check for the expected plain-text rules, an appropriate successful status, redirects and a cache header that explains stale output. An HTML login page, security challenge or 404 document is a different response from the intended file.

For a browser-based check, use our tester to fetch the public file. Compare its returned body with the text you edited. Run it again after publishing rather than relying on a preview.

A public fetch establishes what that request received. It does not prove that Google has already refreshed its cached copy, that every edge location is identical, or that the page is indexed.

A checked before-and-after example

Here is a deliberately broken configuration for a public site:

User-agent: *
Disallow: /
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/wp-sitemap.xml

For this fixture, the intended change is to allow public pages and rendering assets while keeping the admin path disallowed:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/wp-sitemap.xml

These are examples, not a universal production file. Replace the sitemap with your verified URL and retain any deliberate site-specific policies. A store’s filters, search URLs or other special cases need their own review.

On 28 October 2026, we ran both exact bodies through the parser and path-simulation functions used by our tester. We checked the following paths as both Googlebot and Bingbot; both selected the wildcard group and produced the same decisions:

The AJAX path remains allowed in the broken example because its more specific Allow beats the broad Disallow. This is why reading only one line can miss the actual decision.

This was a local rules diagnostic, not an observation of Google or Bing crawling a WordPress installation. It checks these path patterns and group selection; it does not test CDN behavior, crawler refresh times, indexing or every robots extension. Google’s robots specification is the primary reference for Google-specific behavior.

Download the before fixture, after fixture and verification checklist to repeat the comparison with your site’s important paths.

Search visibility is a separate WordPress setting

Settings → Reading → Discourage search engines from indexing this site does not mean “make this website private.” WordPress’s wp_robots_noindex() adds page-level indexing instructions when the site is not public. Core removed the automatic site-wide Disallow: / from do_robots() in WordPress 5.3. Plugins or host rules can still add their own instructions.

If a public site is accidentally discouraged, uncheck the setting after verifying that it is intended to be public. Then inspect an actual page’s rendered robots meta tag and HTTP headers, alongside /robots.txt. Our X-Robots-Tag Checker helps inspect header directives; plugin or server-level rules may still apply after the Reading setting changes.

A robots crawl block cannot reliably remove an indexed URL. It can also stop Google from reading a page’s noindex instruction. Follow Google’s noindex guidance for public pages that should leave search, and require authentication for private content. Robots rules are requests to compliant crawlers, not access controls.

Subdirectories and multisite need an origin-level check

For WordPress installed in a subdirectory, inspect the real admin URLs before copying the root-installation example. Core derives the paths from admin_url(), so /wordpress/wp-admin/ may be appropriate where /wp-admin/ is not.

For multisite under one host, plan root robots rules across the network’s public paths. A subsite administrator’s plugin settings do not by themselves establish which site or routing layer generates the root response. For mapped domains or subdomains, inspect /robots.txt on each relevant public origin. Avoid a shared root file that accidentally blocks another site’s pages.

Keep a representative public page, required stylesheet or script, intended restricted path and sitemap in the acceptance check for each origin. After the edit, fetch the final body, repeat the path tests and inspect an important live page in Search Console when needed. Passing a local rules test does not guarantee indexing or ranking.

Get help fixing the WordPress configuration

Send the public robots.txt URL, the body you intended to publish, the source you edited and one URL with an unexpected decision. Our Website Fix Sprint can cover an agreed configuration priority after assessment. We confirm the scope, price and timeline before starting.

If the underlying maintenance also requires a runtime update, our WordPress PHP update service explains its separate scope. Ongoing WordPress care is available for continued maintenance.

Get help fixing my WordPress robots.txt