WordPress Robots.txt Best Setup: Folders You Should Never Block

A WordPress robots.txt file is a tiny text file that most site owners never open. However, one wrong line can stop Google from loading your theme's stylesheets, and then it judges a broken version of your pages. In this guide, we at Talha Aslan and team explain how WordPress creates this file, which folders you should never block, and what a safe starting setup looks like. If you prefer not to write rules by hand, our free robots.txt generator builds a clean draft in a few clicks.

In other words, a good file is not measured by how much it blocks. It is measured by how few mistakes it contains. In our field work, the most common problem is not a missing rule. Instead, it is an extra rule. Long lists copied from old tutorials often close resources that Google now needs to render a page. Therefore, the rest of this guide explains why each line belongs there. If the basics are new to you, start with what robots.txt is and how to create it. Here, we focus only on the WordPress side.

Why can't you find the WordPress robots.txt file on your server?

Many site owners type /robots.txt after their domain and see content. Then they open the file manager and find nothing. The reason is simple. When there is no physical robots.txt in the root folder, WordPress answers the request itself and generates the content on the fly. The WordPress developer reference shows that a function called do_robots handles this job (official WordPress reference).

So there are two different situations. The first is a virtual file, where WordPress and plugins decide the content in code. The second is a physical file. Once you upload a robots.txt to your server, the web server serves it directly, and WordPress never runs. For example, if a change you made in a plugin does not show up live, an old physical file is probably sitting in the root. Moreover, this happens a lot after migrations and theme switches.

• The URL works but no file exists: you use the virtual file.
• A file exists and plugin settings do not apply: the physical file wins.
• Neither exists and the URL returns 404: check permalinks or server rewrite settings.

Which lines does the virtual WordPress robots.txt include by default?

The default output from WordPress is very lean. The first line is User-agent: *, which addresses all bots. Next comes a Disallow line for the dashboard path and an Allow line that opens admin-ajax.php. WordPress calculates these paths from your admin URL. As a result, if WordPress lives in a subfolder, the path includes that folder too.

In addition, when the built-in WordPress sitemap feature is active, core can add a Sitemap line to this output. If an SEO plugin runs its own sitemap, the plugin usually manages that line. In short, the default output is already a sensible start. In most WordPress robots.txt setups, we keep this core and add only the few lines a site truly needs.

There is one small but important detail. Robots.txt rules are case sensitive. So wp-admin and WP-Admin count as different paths. For that reason, copy the exact form WordPress uses when you write paths by hand.

Why does the admin-ajax.php exception exist?

The dashboard holds nothing useful for bots, so blocking wp-admin makes sense. However, admin-ajax.php lives in that folder and does not serve only administrators. Many themes and plugins use it on public pages. Forms, filters, cart fragments and load more buttons often run through this file.

For example, a product list may fill itself by calling this file after the page loads. If you block the whole folder without an exception, Google cannot make that call while rendering. Then it may miss part of your content. That is why WordPress adds Allow: /wp-admin/admin-ajax.php to the default output on its own. If you write your own file, keep this exception.

So how does Google settle two conflicting rules? We cover the details below. The short answer is that the longer, more specific rule wins. The Allow line describes a longer path, so it overrides the Disallow line for that one file.

Why should you never block wp-content and wp-includes?

Older WordPress tutorials often suggested blocking wp-content and wp-includes entirely. Today that advice is a serious mistake. Your theme stylesheets, fonts, images and the JavaScript that runs interactive parts all live in these folders. Meanwhile, Google evaluates pages by rendering them much like a browser does.

Google's official robots.txt documentation clearly says you should not block resources that make a page harder to understand (Google Search Central). If those resources are blocked, Google may see your page without styles, with a broken layout, or with missing content. As a result, mobile friendliness and content understanding can suffer.

• wp-content/themes: theme styles and scripts live here.
• wp-content/uploads: your images live here; block it and they disappear from image search too.
• wp-content/plugins: front end files of your plugins live here.
• wp-includes: shared WordPress JavaScript libraries live here.

In short, write no Disallow line for these folders. Even if one plugin folder seems to create needless requests, look at the cause first. A blanket block almost always does more harm than good.

How does Google read WordPress robots.txt rules?

Google's robots.txt specification rests on a few core rules (Google robots.txt specification). First, Google picks the User-agent group that matches it best and follows only that group. For example, if the file has both a User-agent: * group and a User-agent: Googlebot group, Googlebot follows only the group that names it.

Next, when Allow and Disallow rules conflict for the same URL, the rule with the longest path wins. If both paths have equal length, Google picks the least restrictive one, which is Allow. Google also applies a size limit to the file and caches it for a period of time. Check the current limit and cache period in Google's documentation.

• The most specific group wins: a separate Googlebot group means the general rules no longer apply to it.
• The longest path wins: this is why the admin-ajax.php exception works.
• Google does not support noindex or crawl-delay lines inside robots.txt.
• Paths are case sensitive.

These rules matter most when you open separate bot groups. The most common mistake we see is a separate Googlebot group that forgets the wp-admin rule.

Which WordPress robots.txt lines are safe to use?

The table below sums up the lines you can safely use on a standard WordPress site, plus the ones to avoid. Treat it as a checklist, since every site has its own needs.

The domain in the table is only an example. Use your own domain and the real address of your sitemap. In the Sitemap line, always use a full URL with the protocol, not a relative path.

Should you block search results and tag pages with robots.txt?

WordPress internal search results, tag archives and thin date archives may be pages you do not want indexed. However, blocking them in robots.txt does not give you the result you expect. Robots.txt stops crawling, not indexing. Google can still index a blocked URL that earns links from other sites, even without seeing its content.

What's more, a blocked page hides its noindex tag from Google. So the right way to remove a page from the index is to keep it crawlable and add noindex. We explain the difference in detail in our noindex vs nofollow and robots.txt comparison.

1. Internal search results: set noindex with your SEO plugin.
2. Tag archives: set noindex if they add no value, or simplify how you use tags.
3. Attachment pages: redirect them to the parent file.

For attachment pages, our guide on WordPress attachment pages and SEO goes deeper. Keep robots.txt for its real job, which is managing crawl activity.

What does the "Discourage search engines from indexing this site" setting do?

This checkbox in the WordPress Reading settings often gets mixed up with robots.txt. According to the WordPress reference, older versions added a line that closed the whole site in the virtual robots.txt when the box was checked. Newer versions removed that behavior. Instead, WordPress now adds a noindex value through the robots meta tag.

So when the box is checked, your robots.txt may look harmless while your entire site carries noindex. This happens a lot when a staging site goes live and nobody unchecks the box. If that sounds familiar, follow the steps in our post on fixing the discourage search engines setting left on.

Therefore, when you audit a WordPress robots.txt setup, look beyond the file. Check the Reading settings and the robots meta tag in the page source too. If the file is clean and the box is checked, the problem is not the file.

How should you use your SEO plugin's robots.txt editor?

Many popular SEO plugins let you edit robots.txt from the dashboard. These editors usually work in one of two ways. Some change the virtual output. Others write a physical file to the root folder. Check your plugin's documentation to learn which one applies.

In practice, the key point is simple: use only one method at a time. For example, if an old physical file sits in the root while you edit in the plugin, your changes never go live. Also, two plugins adding lines to the same output can cause confusing results, such as duplicate Sitemap lines.

1. Save a copy of the current output before you change anything.
2. Write only the lines you actually need in the editor.
3. After saving, open the live URL in a browser and confirm the result.

Should you use a physical file or the virtual file?

Both have their place. The virtual file is easy to maintain, since WordPress calculates paths for you and plugins handle the Sitemap line. A physical file gives you full control. However, you must update it by hand after a migration or a subfolder change.

For most small and mid sized sites, we recommend the virtual file. A physical file starts to make sense for multilingual setups, complex filters or custom bot rules. Either way, keep one single source of truth.

How do you set up WordPress robots.txt step by step?

Follow this order and you get a file that is both safe and easy to maintain. We designed these steps so you can move forward without breaking your site.

1. Add /robots.txt to your domain, open the current output and save a copy.
2. Check the root folder in your file manager for a physical robots.txt.
3. Make sure the discourage search engines box in Reading settings is unchecked.
4. Draft the new content with our robots.txt generator tool, including the wp-admin rule, the admin-ajax.php exception and your sitemap URL.
5. Apply the content in one place only: either your SEO plugin's editor or the physical file.
6. Reopen the live URL and confirm the new lines appear.
7. Use the robots.txt report in Search Console to see how Google reads the file.

For the last step, our guide on how to read the Search Console robots.txt report walks you through it. That way, you confirm Google sees the change correctly too.

How do you add the Sitemap line to a WordPress robots.txt file?

The Sitemap line tells Google where your sitemap lives. It does not belong to any User-agent group, so it can sit anywhere in the file. Still, we usually put it at the bottom for readability. The address must be a full URL, which means it includes the protocol and the domain.

If both the built-in WordPress sitemap and your SEO plugin's sitemap run at the same time, decide which one you use. Then list only that one in robots.txt. If you have several sitemaps, you can write several Sitemap lines. If you have none, or want to rebuild it, our XML sitemap generator makes the job easier.

That said, a Sitemap line alone does not guarantee indexing. It only helps Google discover your URLs. So it is a good habit to submit the sitemap in Search Console as well.

Which WordPress URLs should never appear in robots.txt?

Some lines look like security measures at first glance but actually cause harm. For example, listing your login page or a hidden admin URL in robots.txt announces that address to everyone. Robots.txt is a public file, and malicious bots ignore its rules anyway.

• Login or admin URLs you want to keep private.
• The wp-content, wp-includes and uploads folders.
• Pages you want removed from the index that already carry noindex.
• Duplicate URLs you already handle with canonical tags.

So for security, rely on strong passwords, two step verification and updated plugins, not robots.txt. For duplicate content and parameter URLs, a canonical tag is often the better tool. For special cases such as filter pages, see our post on faceted navigation and robots.txt.

When should you use wildcards in WordPress?

Google supports the asterisk and the dollar sign in robots.txt rules. The asterisk matches any sequence of characters. The dollar sign marks the end of a URL. On WordPress, these are most useful to control URLs that generate endless parameters. For example, if sorting or filter parameters create thousands of URLs, you may not want crawlers to spend time there.

However, a badly written wildcard can block a far wider area than you intended. Therefore, test every wildcard rule against a few real URLs after you write it. For detailed examples, our robots.txt wildcards guide can help.

In short, a standard business WordPress site often needs no wildcard at all. On ecommerce sites and filtered catalogs, use them with care and in moderation.

How should you test robots.txt after a change?

If you skip testing, a small typo can go unnoticed for weeks. First, open the live URL in your browser and confirm the file returns plain text with the lines you expect. Next, check the robots.txt report in Search Console to see which version Google fetched last and whether it found errors.

Then use the URL Inspection tool on a few key pages. Confirm that stylesheets load in the rendered view. That way, you spot right away whether you blocked CSS or JavaScript by mistake. Also, use our post on common robots.txt mistakes and fixes as a checklist.

• Does the live URL return plain text?
• Is admin-ajax.php open?
• Do theme files load in the rendered page?
• Does the Sitemap line point to the right URL?

Do AI crawlers need separate rules?

Some site owners do not want their content used to train AI models. Several of these crawlers state that they follow robots.txt under their own User-agent names. So, if you wish, you can add separate groups for these bots to your WordPress robots.txt file.

However, the rule from earlier matters again here. A bot with its own group does not follow the rules of the general group. So when you open a separate group, repeat the rules you need inside it. We explain which bot does what, and how to tell them apart, in our post on AI crawlers and robots.txt. When you decide, take care not to block bots that affect your search visibility.

What changes for multilingual or subfolder installs?

Robots.txt only works at the root of a host. So even if WordPress runs in a subfolder, Google looks for the file at the domain root, not in the subfolder. In that case, the root file must point to the correct WordPress paths.

Also, each subdomain has its own robots.txt. For example, if your blog sits on a subdomain and your store on the main domain, you manage two separate files. On multilingual sites, blocking language folders is almost never right. Handle language versions with hreflang tags instead. In short, check the root of every host separately.

On staging and test sites, the situation flips. Closing the whole site there can make sense. Just remember to remove that line at launch. This mistake is one of the most common causes of traffic loss after a migration.

Does a WooCommerce store need a different WordPress robots.txt?

On a WooCommerce site, the core setup does not change. wp-admin stays blocked, admin-ajax.php stays open, and theme files stay reachable. However, stores generate many more dynamic URLs. Cart, checkout and account pages are personal to each visitor and add no value in search results. Moreover, add to cart links create parameter URLs that can waste crawl activity.

So for stores, we weigh two approaches together. First, use noindex on personal pages such as cart and checkout; many SEO plugins already do this. Second, write a measured Disallow rule only for parameters that create endless URLs. For example, blocking URLs that contain the add to cart parameter cuts needless crawling without touching product pages.

On the other hand, never block product images or category pages. These pages carry your store's search traffic. In short, the rule stays the same for stores: keep everything that serves visitors open, and limit only URLs that create endless copies.

How does a robots.txt mistake hurt traffic, and how do you notice it?

The sneaky part of robots.txt mistakes is that the site looks perfectly normal to visitors. Pages open and forms work. Meanwhile, Google cannot see the pages in the background. So you usually notice the signs in Search Console data. For example, impressions may drop sharply within days, or the page indexing report may show more URLs blocked by robots.txt.

Another sign is search results without a description. When Google keeps a page indexed but cannot see its content, it may show a similar notice that no information is available instead of a snippet. When you see that pattern, check your robots.txt file and its last change date first.

• Sudden, unexplained drops in impressions.
• A rise in robots.txt related blocks in the indexing report.
• An unstyled screen in the rendered page view.
• Search results that show no description.

Therefore, add the robots.txt output to your checklist after every major theme, plugin or server change. This simple habit helps you catch a visibility loss early, before it lasts for weeks.

Can old lines in robots.txt cause problems?

On WordPress sites that have been live for years, robots.txt often grows layer by layer. Every new plugin, consultant or forum tip adds a line, and nobody removes the old ones. Eventually, the file collects rules for folders that no longer exist, typos and lines Google does not support.

For example, Google ignores the crawl-delay line, and it does not support a noindex line inside robots.txt either. These lines look harmless, yet they give you a false sense of control. In addition, a rule that blocks the folder of a retired plugin may one day block a new structure with the same name.

So review the file line by line at least once a year. For each line, ask yourself a simple question: what problem does this rule solve today? If you cannot answer, remove the line and watch the result in Search Console. Also, keep a short change log with dates, especially if several people work on the site.

Quick checklist: a safe starting setup

If we boil this guide down to one list, the following setup is a safe start for a standard WordPress robots.txt file on most sites. Weigh each item against the needs of your own site.

1. Open one general group for all bots.
2. Block the wp-admin path and keep the admin-ajax.php exception.
3. Leave wp-content and wp-includes alone.
4. List your sitemap with a full URL.
5. Use noindex for pages you want out of the index.
6. Test with Search Console after every change.

If you want to tackle this and other parts of technical SEO together, our SEO consulting service reviews your site's crawling and indexing setup end to end. Still, for most WordPress sites, the short setup above plus a regular testing habit will be enough.

Core guide: What Is Digital Marketing? Its types, examples and how it is done, on one page with an interactive film that travels through the brain.