Search Console Robots Report: How to Read It and Fix robots.txt Errors

If you want to know how Google actually sees your robots.txt file, the Search Console robots report (officially the robots.txt report) is the first place to look. In this guide, Talha Aslan and team walk through the report line by line. We explain what each status means and the order in which you should fix problems. If you do not have a file yet, you can draft a clean one with our free robots.txt generator first.

However, the report does not edit anything. It only tells you what Google received. For that reason, you always make changes in the file on your server. Then you use the report to confirm that the change reached Google.

If robots.txt is new to you, we cover the basic syntax in a separate article: what robots.txt is and how to create it. In this article, we therefore focus only on reading the report.

The real value of checking the report is early warning. For example, a theme update, a security plugin or a server change can quietly break the robots.txt response. If you spot it before traffic drops, the fix usually takes minutes.

What happened to the old robots.txt tester?

The older version of Search Console had a separate tester. You could paste rules into it and check whether a URL was blocked. Today, Google's official guide to creating a robots.txt file lists two options for fixing markup issues. The first is the robots.txt report in Search Console. The second is Google's open source robots.txt library.

Still, do not skip one detail in that guide. The report only works for files that are already accessible on your site. So if you want to test a draft before it goes live, you need local testing or a careful preparation process instead.

For example, a developer on your team can build the open source library and test rules locally. If you have no developer support, keeping the rules simple is the safest path.

Our advice is to change one small set of rules at a time. After publishing, open the report and confirm the result. As a result, you know which line caused a problem. Also, keep a copy of the previous version so you can roll back in minutes.

Where do you find the report in Search Console?

In practice, the report sits in the settings area of Search Console. Menu labels can change over time. So we do not recommend relying on exact button text. When you open settings, you will see a robots.txt entry in the crawling section.

1. Sign in to Search Console and pick the right property.
2. Open the settings area from the left menu.
3. Click the robots.txt entry under the crawling heading.
4. Review the table, which lists a separate row for each host.

The property type matters here too. A domain property can list several hosts in the same report, such as the www version, the non-www version and subdomains. A URL prefix property only covers that prefix. If you are setting up Search Console for the first time, our Google Search Console guide is a good starting point.

Which columns does the report show and what do they tell you?

The table is simple. However, each column answers a different question. Keep the table below open next to the report.

Next, when you click a row, you see the file content that Google last fetched. The report also lists versions fetched in the recent past. This lets you track the day a change reached Google.

What do the fetch status labels mean?

You will meet three main outcomes in the report. In addition, the exact wording can vary slightly with your interface language. So below we explain the meaning rather than the exact label.

• Fetched: Google retrieved the file. Check the issues column separately.
• Not found (404): The server said there is no file. Google then assumes there are no crawl restrictions.
• Not fetched for another reason: A server error, timeout or network problem got in the way. This is the most critical case.

For example, a 404 often needs no panic. But if your site intends to block certain sections, a 404 means none of those rules apply. As a result, folders you wanted to keep out of crawling stay open.

Also, a 404 sometimes comes from the wrong location. If you uploaded the file to a subfolder, Google will not look there. Google only reads the file in the root directory, right after the domain name.

How does Google crawl your site when the status is 404?

According to Google's robots.txt specification, Google treats 4xx responses as if no robots.txt file exists. In other words, Google assumes there are no crawl restrictions. The document excludes the 429 code from this rule.

The same document also says you should not use 401 or 403 to limit crawl rate. So if your security plugin returns a 403 to Googlebot, do not see it as a restriction method. Google reads that response as a missing file.

In practice, you open the file URL in a private browser window. If the content appears, the problem is likely a cache or a redirect. If nothing loads, you upload the file to the root directory again.

What does Google do when it sees a server error (5xx)?

This is the most important part of the report, because a 5xx status directly affects crawling. Google's official documentation describes a staged process.

1. For the first 12 hours, Google stops crawling the site but keeps requesting the file.
2. Then, for up to 30 days, Google uses the last good cached version if it has one. Without a cached copy, it assumes no restrictions.
3. After 30 days, if the site is generally available, Google acts as if there is no file. If the site is unavailable, Google stops crawling and requests the file periodically.

In short, a persistent server error on the robots.txt URL can slow crawling even when your pages are healthy. If you see a clear drop in your crawl chart, the steps in why Googlebot crawls less will help with the diagnosis.

How does the report show a robots.txt file that redirects?

Google's documentation says Googlebot follows at least five redirect hops for robots.txt. If it still cannot reach the file, Google treats the result as a 404. Also, Google does not follow logical redirects such as JavaScript or meta refresh.

So a single redirect from HTTP to HTTPS, or from www to non-www, causes no trouble. On the other hand, chained redirects from plugins, a CDN and the server can add up quickly.

In practice, if you see an unexpected 404 in the report, open the file URL in a header checker and count the hops. If the chain is long, point the redirect straight to the final URL. That helps both robots.txt and overall crawl efficiency.

How should you read warnings in the issues column?

If the file was fetched but the issues column shows a number, Google could not understand some lines. According to the official documentation, Google ignores invalid lines. A byte order mark (BOM) at the start of the file falls into the same group.

In our experience, these are the most common causes:

• Misspelled directive names, such as a Disallow line without its colon.
• Rule groups that start without a User-agent line.
• Invisible characters pasted from a word processor.
• A file saved in an encoding other than UTF-8.

Warnings do not cancel the whole file. Still, they can mean that a block you expected is not working. Therefore, open each warning and fix the matching line. Our article on common robots.txt mistakes lists frequent syntax errors in detail.

Why does the file size limit matter in the report?

Google's documentation sets a 500 KiB size limit for robots.txt and says Google ignores content beyond it. That is exactly why the size column is useful.

In addition, most business sites have a file of a few lines, far from the limit. However, on large e-commerce sites, the file can grow out of control when someone adds a line for every filter combination. In a file over the limit, the rules at the bottom quietly stop working.

The fix is usually to merge rules. For example, you write one pattern with a wildcard instead of dozens of similar lines. This keeps your file short and easy to read. We cover strategy for filtered URLs in our article on faceted navigation.

How do rules in the file match URLs?

When you open the file content in the report, you need to work out which line affects which URL. Google's documentation defines a few simple principles, and they clear up most confusion.

• The asterisk (*) matches zero or more characters.
• The dollar sign ($) marks the end of the URL.
• Google ignores a trailing asterisk.
• Path matching is case sensitive.

Example scenario: your file has Disallow: /search and Allow: /search/guide. For the URL /search/guide, the longer match is the Allow line. Therefore Google can crawl that URL. On the other hand, /Search starts with a capital letter, so it matches neither rule.

In short, these small details explain why some pages are not crawled while the report looks fine. If you keep your rules simple, matching also becomes easier to predict.

How do the cache and the check date relate?

The check date confuses people more than any other column. According to Google's documentation, Google generally caches robots.txt content for up to 24 hours. This period can be longer during timeouts or server errors. Also, the max-age value in your Cache-Control header can affect the cache lifetime.

In other words, the check date tells you the age of the version Google holds. If you fixed the file this morning and the date shows yesterday, Google is still crawling with the old rules.

In practice, you follow this order. First you fix the file, then you confirm the new content in a browser, and then you wait for the date to change. In an urgent case, a recrawl request can shorten the wait.

For example, if you use a CDN, remember that the CDN may also cache an old copy. In that case you clear the CDN cache too. Otherwise Google keeps receiving the old version.

When should you send a recrawl request?

Because Google usually caches robots.txt for up to 24 hours, a fix can take a day to reach Google. Google's guide on updating your robots.txt file recommends the recrawl request function of the report when you need to refresh the cache faster.

That said, you do not need it for every small change. We use it in these cases:

• You fixed a rule that blocked the whole site by mistake.
• You opened an important section to crawling.
• A long server error has just been resolved.

Do not treat this function as a magic button. Google reviews the request and follows its own crawl schedule. For routine, non urgent updates, waiting for the cache to refresh is enough.

Before you send the request, confirm that the new file is live. Otherwise Google fetches the old content again, and you may think the fix failed. The order is always file first, browser check second, request last.

How do you check whether a single URL is blocked?

The Search Console robots report shows the file itself. To see whether a specific page is blocked, you use the URL Inspection tool. In short, the two tools complement each other.

1. Type the full URL into the search box at the top of Search Console.
2. Open the crawl details in the result.
3. Read the line that states whether crawling is allowed.
4. If you see a block, open the matching host's file in the robots.txt report and find the rule.

Remember Google's precedence logic while you search. The longest matching path wins. If an Allow and a Disallow rule of equal length conflict, Google applies the less restrictive one.

Why does a blocked page still appear in Google?

For example, this surprises many people who read the report. Google's introduction to robots.txt states it clearly: robots.txt is not a way to keep a page out of Google. A blocked page that gets links from other sites can still end up in the index.

As a result, such a page may show in results without a description. So if your goal is to remove a page from search, you use noindex or protect the page with a password.

However, there is a trap here. Google must crawl the page to see a noindex tag. If you block a page and add noindex at the same time, Google can never read the tag. We compare the two methods in our noindex, nofollow and robots.txt comparison.

How do subdomains and protocols show up in the report?

According to Google's documentation, robots.txt rules apply only to the host, protocol and port where the file lives. A file for blog.example.com does not affect example.com. Google also handles the HTTP and HTTPS versions separately.

So you read each row in the report as a separate file. For example, the main site's row can look clean while an old staging subdomain shows a 404 or a server error.

If such a subdomain is publicly reachable, the right way to protect it is a password or access restriction. Robots.txt alone is not enough because, as we just saw, blocked URLs can still get indexed.

How can you use the Search Console robots report with server logs?

The Search Console robots report summarizes Google's latest fetch. Server logs, by contrast, show every single request. Reading both is the most reliable way to catch intermittent errors.

For example, a single server error in the report may be a one time glitch. But if the logs show frequent 5xx codes or timeouts for Googlebot's robots.txt requests, the problem is persistent.

1. Get the access logs from your hosting panel or your technical team.
2. Filter the requests for the robots.txt URL.
3. Compare response codes and response times.
4. Match error times with traffic peaks or maintenance windows.

In addition, this tells you whether your firewall blocks Googlebot by mistake. Sharing exact times from the logs with your hosting company speeds up the fix. Keep in mind that a user agent name can be faked. For suspicious traffic, see our guide on how to verify Googlebot.

Where does robots.txt come from on hosted platforms?

However, the content in the report may differ from the file you uploaded. Some content management systems generate robots.txt themselves when no physical file exists in the root. For example, WordPress serves a virtual file in that case, and SEO plugins can change its content.

So when the content in the report surprises you, find the source first. Is there a physical file in the root, or does the system generate it? Does a plugin setting conflict with a physical file?

Likewise, hosted e-commerce platforms and site builders often serve a default file and limit editing. On such a platform, you can only edit what the platform allows. Therefore, check the report after every change to confirm what Google actually receives.

Once the source is clear, manage the file from one place. Two sources mean a new surprise with every update.

What should you check in the report after a site migration?

Above all, domain changes, HTTPS moves and platform migrations are when robots.txt errors appear most often. On migration day, open the report and look for answers to these questions.

• Does Google fetch the new host's file, or does it return a 404 or a server error?
• Did the staging rule that blocks everything make it to production?
• Does the Sitemap line point to the new domain?
• Can Google still reach the file on the old domain without breaking redirects?

Example scenario: after a migration, a single Disallow: / line stays live, and Google cannot crawl the new pages. A team that checks the report on day one fixes it within hours. To diagnose a traffic drop after a move, see our guide on diagnosing traffic drops in Search Console.

How do you fix an error you found in the report, step by step?

After that, the fix order is the same on most sites. Our team uses this flow in technical audits.

1. Note the affected host and the fetch status in the report.
2. Open the file URL in a private window and see what the server returns.
3. If needed, rebuild the file with the robots.txt generator and add only the paths you truly want to block.
4. Upload the file to the root directory as UTF-8 plain text.
5. Confirm that the new content appears in the browser.
6. For urgent fixes, send a recrawl request from the report.
7. Check a few key URLs with the URL Inspection tool.

If you add a Sitemap line to the file, make sure Google can read the sitemap too. If you see a sitemap error, our article on the sitemap could not be read error will guide you next.

How do you confirm the effect of a robots.txt change?

Fixing the file is only half the job. The other half is seeing that the effect really happened. The report only shows the state of the file, so you look at several screens together.

1. Confirm that the check date moved past the time of your change.
2. Open the file content and confirm that the new rules appear in Google's copy.
3. Recheck a few important URLs that were blocked before in the URL Inspection tool.
4. Over the next days, watch the change in request counts in the crawl stats.

Example scenario: you removed a rule that blocked category pages by mistake. Once the check date updates, URL Inspection shows that crawling is allowed for those pages. Their status in the indexing report changes over time. However, this depends on Google's crawl queue, and there is no fixed timeline.

Patience matters at this stage. On large sites, recrawling every reopened section can take days or even weeks, and that is not a sign of an error. Additionally, we suggest logging each change with its date. Then, weeks later, you know exactly which rule changed and when.

Which mistakes should you avoid when reading the report?

The report looks simple, yet people misread it often. These are the mistakes we see most in the field.

First, assuming a successful fetch means everything is fine. Google may have the file, but a single Disallow line inside it may block an important folder. That is why you read the content along with the status.

Second, expecting the report to update minutes after a change. Because of the cache, the new content will not show until the check date changes.

Third, thinking the report shows only one host. With a domain property, you need to open several rows one by one. Finally, do not always look for indexing problems in robots.txt. For other causes, see our guide on finding unindexed pages in Search Console.

How often should you check the Search Console robots report?

The Search Console robots report does not need daily attention. We suggest opening it once after every site release, theme change, CDN change or security plugin update.

In addition, you can add it to your monthly technical checklist. This way, an unnoticed server error or a test file pushed live by mistake does not linger for long.

Example scenario: after a migration, the staging file that blocks everything gets copied to production. A team that opens the report on migration day sees this within hours. A team that does not notices the crawl drop weeks later.

When should you get expert help with robots.txt and crawling issues?

In most cases, a simple file needs no expert. However, the problem may go beyond robots.txt if the report keeps showing server errors, if filter URLs drain crawling on a large store, or if traffic falls after a migration.

In those cases you need to look at crawling, indexing and site architecture together. Within our SEO consulting work, we review all three in a single technical audit. We do not promise results, but we aim to show where the problem is with evidence.

Before you contact an expert, prepare a screenshot of the report, the current file content and the date the problem started. Together, these three items speed up the first diagnosis a lot. Moreover, a list of recent technical changes often points to the cause as well.

If you prefer to handle it yourself, keep your sitemap current with our XML sitemap generator alongside robots.txt. Together they help Google understand your site correctly.

Core guide: What Is Digital Marketing? Its types, examples and how it is done, on one page with an interactive film that travels through the brain.