WordPress robots.txt: What to Put in It and Why

Most WordPress sites do not have a robots.txt file at all. Visit https://yoursite.com/robots.txt and you will still see one, because WordPress generates it on the fly. That gap between “the file you see” and “the file that exists” is where most WordPress robots.txt problems start: people edit a physical file that a plugin then ignores, or a plugin setting that a physical file silently overrides.

This guide covers how the WordPress robots.txt actually works, how to edit it safely, what a sensible file looks like for a typical site, and the rules we would remove from almost every WordPress robots.txt we see.

How WordPress Generates robots.txt

WordPress core includes a function called do_robots(). When a request for /robots.txt reaches WordPress and there is no physical robots.txt file in the web root, WordPress builds a virtual one and serves it.

On a current WordPress install with search engine visibility enabled, the virtual file looks like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml

Three things are going on:

1. Disallow: /wp-admin/ keeps crawlers out of the admin area, which has nothing worth indexing.
2. Allow: /wp-admin/admin-ajax.php re-opens the one admin endpoint that themes and plugins call from the front end. Blocking it can stop Google from rendering content that is loaded via AJAX.
3. Sitemap: points to the core XML sitemap that WordPress has shipped since version 5.5. If an SEO plugin replaces the core sitemap, the plugin usually swaps this line for its own sitemap index (for example sitemap_index.xml).

Because the file is virtual, plugins and themes can change it through the robots_txt filter without touching the file system. That is exactly how Yoast SEO and Rank Math add their sitemap lines.

The physical file always wins

If you upload a real robots.txt to the web root, your web server serves it directly and WordPress never runs do_robots(). Any plugin setting that edits the virtual file stops having an effect. This is the single most common reason someone says “I changed robots.txt in my SEO plugin but nothing happened.”

Quick check: open the file over SFTP or your host’s file manager. If robots.txt exists in the same folder as wp-config.php, it is the one being served.

How to Edit robots.txt in WordPress

There are three reliable ways. Pick one and stick with it, so there is only ever one source of truth.

Option 1: Yoast SEO

1. Go to Yoast SEO → Tools → File editor.
2. If no physical file exists, Yoast offers to create one. Click Create robots.txt file.
3. Edit and save.

Note that Yoast’s file editor writes a physical file. From that point on, the virtual WordPress file is no longer used. If the File editor option is missing, file editing is probably disabled on your install (for example via DISALLOW_FILE_EDIT in wp-config.php) or your host does not allow PHP to write to the web root.

Option 2: Rank Math

1. Go to Rank Math → General Settings → Edit robots.txt.
2. Edit the content and save.

Rank Math edits the virtual file through the filter. If a physical robots.txt exists, Rank Math shows the field as read-only and tells you to delete the physical file first.

Option 3: Upload the file yourself

1. Create a plain-text file named robots.txt (UTF-8, no BOM).
2. Upload it via SFTP or your host’s file manager to the web root.
3. Load https://yoursite.com/robots.txt in a private window to confirm the new content is served. Clear any page cache or CDN cache if you still see the old version.

This option is the most predictable, especially on managed hosts and multisite setups, because no plugin can change it behind your back.

A Recommended WordPress robots.txt

For a typical WordPress blog or business site, this is close to what we recommend:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/

Sitemap: https://example.com/sitemap_index.xml

Why each line is there:

• Admin rules are the WordPress defaults and are safe.
• /?s= and /search/ block internal search result URLs. Internal search pages can generate an unlimited number of thin, near-duplicate URLs, and they are a common source of spam: bots submit queries containing spammy phrases, and those URLs can end up discovered and crawled. Blocking them saves crawl activity for pages that matter.
• Sitemap: should point to whichever sitemap your site actually serves. Open it in a browser to confirm it returns XML, not a 404. If you are unsure which sitemap that is, our guide on how to find the sitemap of a website walks through it.

WooCommerce additions

Stores often add:

Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*?add-to-cart=

These pages are user-specific and have no search value. The add-to-cart rule matters more than people expect: every product listing link with ?add-to-cart= is a crawlable URL that triggers server work.

Faceted filters (?filter_color=, ?orderby=) are a judgment call. On small catalogs, leave them crawlable and handle duplicates with canonicals. On large catalogs where filters multiply into thousands of combinations, blocking the worst parameter patterns can help. Our faceted navigation guide covers that decision in detail.

What Not to Put in a WordPress robots.txt

Here is the angle most robots.txt tutorials get wrong: they treat the file as a list of “things to hide.” Most WordPress robots.txt problems come from rules added with good intentions years ago.

Do not block /wp-content/ or /wp-includes/

Old guides recommended this to “protect” WordPress folders. Those folders hold the CSS, JavaScript and images Google needs to render your pages. Google renders pages much like a browser, so blocking these resources can make pages look broken or empty to Googlebot. Google’s own documentation warns against blocking resources needed for rendering.

Do not block /wp-content/uploads/

That is where your images live. Block it and your images disappear from Google Images, and pages that rely on them for context lose signals.

Do not use robots.txt to remove pages from Google

Disallow stops crawling, not indexing. If other sites link to a blocked URL, Google can still index the URL itself, typically with no description, because it cannot read the page. Worse, a noindex tag on a blocked page is never seen, because Google is not allowed to fetch it. To remove tag archives, attachment pages or thin pages from the index, use noindex (via Yoast or Rank Math) and leave them crawlable until they drop out.

Do not block feeds, tags or categories by reflex

Feeds help discovery. Category pages are often among a blog’s strongest hub pages. If tag archives are thin, noindex them; do not wall them off in robots.txt.

Be careful with blanket bot blocks

Since generative AI crawlers appeared, many site owners have added blocks such as:

User-agent: GPTBot
Disallow: /

That is a legitimate business choice. Just be precise about which user agents you block. Blocking Googlebot or adding User-agent: * / Disallow: / while trying to block AI bots will remove you from search entirely. Google’s AI training opt-out uses a separate token, Google-Extended, which does not affect Google Search crawling.

The “Discourage Search Engines” Checkbox

Under Settings → Reading, WordPress has a “Discourage search engines from indexing this site” checkbox. It is meant for staging and development sites. Current WordPress versions implement it mainly with a noindex robots meta tag rather than relying on robots.txt.

The classic failure mode: a site is built on staging with this box checked, migrated to production, and launched with the box still checked. Traffic falls off a cliff a few weeks later. Add “Search engine visibility setting” to every launch checklist, and verify it on the live site as part of a site migration checklist.

How to Test Your WordPress robots.txt

1. Fetch it directly at https://yoursite.com/robots.txt and confirm it returns status 200 with text/plain content. A 5xx error on robots.txt is serious: Google may treat the whole site as temporarily off limits.
2. Check the robots.txt report in Google Search Console (under Settings). It shows which robots.txt files Google found, when it last fetched them, and any parsing problems.
3. Test important URLs with the URL Inspection tool to confirm they are not blocked.
4. Validate the syntax with a robots.txt validator before deploying changes.
5. Re-check after plugin changes. Switching SEO plugins often changes the sitemap line, and the old URL may 404.

A Worked Example

Here is an inherited robots.txt we would rewrite on sight (illustrative, but very representative):

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/
Disallow: /tag/
Disallow: /feed/
Disallow: /*?*

Sitemap: https://example.com/sitemap.xml

Problems, in order of severity:

1. /wp-includes/, /wp-content/plugins/ and /wp-content/themes/ block CSS and JavaScript Google needs to render the page.
2. /*?* blocks every URL with a query string, including legitimate paginated or tracked URLs and admin-ajax.php calls with parameters.
3. /tag/ blocks tag archives but they remain indexed via links, now with no content Google can read. noindex is the right tool.
4. /feed/ removes a useful discovery path for no benefit.
5. The sitemap URL points to sitemap.xml, but the site runs Yoast, which serves sitemap_index.xml.

The fixed version is the recommended file shown earlier, plus noindex on tag archives set in the SEO plugin.

For a wider look at directives, wildcards and crawl budget, see our robots.txt best practices guide and the WordPress technical SEO audit.

Frequently asked questions

Where is the robots.txt file in WordPress?

On most installs there is no physical file. WordPress generates a virtual robots.txt when /robots.txt is requested. If a real file exists, it sits in the web root, in the same folder as wp-config.php, and it overrides the virtual one.

How do I edit robots.txt in WordPress without a plugin?

Create a plain-text file named robots.txt, upload it to your web root via SFTP or your host’s file manager, and clear any page or CDN cache. The physical file replaces the WordPress virtual file immediately.

Why are my Yoast or Rank Math robots.txt changes not showing?

Usually because a physical robots.txt file exists and the server serves it instead of the plugin’s virtual version, or because a cache is still serving the old file. Check the web root and purge caches.

Should I block wp-content in robots.txt?

No. wp-content contains the CSS, JavaScript and images Google needs to render your pages. Blocking it can make pages look broken to Googlebot and removes images from Google Images.

Can robots.txt remove a WordPress page from Google?

No. Disallow only stops crawling. To remove a page from search results, add a noindex directive and keep the page crawlable so Google can see it.