Tools
Robots.txt Generator
Build a custom robots.txt file for your site in seconds: ready-made templates, a line-by-line rule editor, AI bot blocking, sitemap support and a URL tester. Free, no sign-up.
Save the output as robots.txt and upload it to your root directory; it must open at https://yoursite.com/robots.txt. Files in subdirectories are ignored, and each subdomain uses its own file.
Result
Type a path to see the result under your rules instantly.
- Digital Marketing Expert
- Google Partner
- Full Stack Developer
- Last updated
- Based on
- 5 sources
How to use the Robots.txt Generator
- Start from a template
Pick Allow all, Block all, WordPress or WooCommerce in the top bar. The template fills in the rule rows, and you can then edit them however you like.
- Write your rules line by line
Enter a user-agent, a rule type and a path on each row. The tool merges rows for the same bot into one group and adds a missing leading slash for you.
- Choose which AI bots to block
Tick the bots you want to keep out. The tool writes them into a separate group with Disallow: /, and the labels tell you which ones train models and which ones power search.
- Add your sitemap URLs
Type one full URL per line. Each one then becomes its own Sitemap line at the end of the file, and a relative path shows up as a critical error.
- Test real URLs
Type a path and a bot name in the tester. The badge tells you whether that bot may crawl the URL, and the line below names the rule that decided it.
- Copy or download
Use Copy or Download, upload the file to your root directory, then confirm it in the Search Console robots.txt report.
How the tool evaluates your rules
The tester follows the matching logic that Google documents and RFC 9309 standardizes. In practice, it reaches a verdict in three steps: first the group, then the rule, finally the tie break.
A bot follows the group that names it; if none exists, it follows User-agent: *, and with no group at all everything is allowedAmong the rules that match the URL, the one with the longest path wins; the order of lines in the file does not matterIf an Allow and a Disallow rule have the same length, Allow wins* stands for zero or more characters: /*.pdf matches any URL that contains .pdf$ pins the end of the path: /*.pdf$ only matches URLs that end in .pdfA Disallow: line without a path blocks nothing, and /robots.txt itself is always readablePaths are case sensitive, user-agent names are not. Other bots may read the syntax differently; the tester result applies to Google and to crawlers that follow the standard.
Example rules and test results
These rows are example scenarios; type the same rules into the tool and the same path into the tester, and you will see exactly the same result.
| Rules | Bot | Tested path | Result |
|---|---|---|---|
| Disallow: /wp-admin/ and Allow: /wp-admin/admin-ajax.php | Googlebot | /wp-admin/admin-ajax.php | Crawlable: Allow is longer |
| Disallow: /wp-admin/ and Allow: /wp-admin/admin-ajax.php | Googlebot | /wp-admin/options.php | Blocked: Disallow: /wp-admin/ |
| Disallow: /shop | Googlebot | /shopping-guide/ | Blocked: prefix match |
| Disallow: /*.pdf$ | Googlebot | /catalog.pdf?v=2 | Crawlable: URL does not end in .pdf |
| User-agent: GPTBot and Disallow: / | Googlebot | /blog/ | Crawlable: no group for Googlebot |
| User-agent: GPTBot and Disallow: / | GPTBot | /blog/ | Blocked: Disallow: / |
| Disallow: /Blog/ | Googlebot | /blog/post/ | Crawlable: case sensitivity |
Unless stated otherwise, the rules sit in the User-agent: * group. The first word in the Result column is what the tester badge shows.
AI bots and what blocking them changes
What the six bots in the tool do according to their official documentation, and what changes when you give them Disallow: /:
| User-agent | Company | Purpose | If you block it |
|---|---|---|---|
| GPTBot | OpenAI | Training generative AI foundation models | Your content stays out of training crawls; your ChatGPT search presence stays the same |
| OAI-SearchBot | OpenAI | Surfacing websites in ChatGPT search features | ChatGPT search answers will not show your site |
| ClaudeBot | Anthropic | Model training and development | Your content stays out of Anthropic's training crawls |
| Google-Extended | Gemini training and grounding, meaning answers backed by Search data | Your inclusion and ranking in Google Search stay the same | |
| CCBot | Common Crawl | Building an open web archive | New crawls leave your site out of the archive |
| PerplexityBot | Perplexity | Surfacing websites in Perplexity search; not for model training | You lose the chance to appear as a source in Perplexity answers |
Source: the bot documentation of OpenAI, Anthropic, Google, Common Crawl and Perplexity. Some user triggered fetchers, such as Perplexity-User, generally ignore robots.txt. Bot names and purposes can change, so check the current docs before you decide.
What is a robots txt generator and what does this one build?
A robots.txt generator is a tool that helps you write a clean robots.txt file, the plain text file that tells search engine and AI bots which paths on your site they may crawl. This generator starts from four templates, lets you edit rules line by line and produces the output instantly. It also merges rows for the same user-agent into one group, fixes paths without a leading slash and appends your sitemap URLs at the end.
On the right, you get three helpers. First, the badge and the check list catch a line that closes the whole site, a relative sitemap URL, blocked CSS or JS and duplicate rules. The tester then tells you whether a URL is crawlable for a given bot and which rule made that call. Finally, you can copy the file or download it straight away as robots.txt.
My team and I check this file first on every site launch and every migration we handle. In SEO consulting projects, the most frequent crawl problems we fix start right here. The tool runs in your browser, so your rules never reach a server.
What does robots.txt do, and what can it not do?
The main job of robots.txt is to manage crawler traffic. Google describes its purpose as keeping your site from being overloaded with requests. In practice, that means keeping bots away from low value URLs such as carts, internal search, filters and admin screens. As a result, bots spend their time on the pages you actually want to rank.
However, the file has three hard limits:
- It does not deindex. A URL you block can still appear in results without a description if other sites link to it. To keep a page out of the index, you need noindex.
- It is not security. Anyone can read the file; moreover, malicious bots ignore it. RFC 9309 also states plainly that these rules are not a form of access authorization.
- Not every bot reads it the same way. Google and standard compliant crawlers agree on the syntax, but some bots behave differently.
So think of robots.txt as directions you give to bots, not as a lock. You get the real benefit when you handle crawling together with the rest of your technical SEO.
Robots txt generator presets for WordPress and WooCommerce
The WordPress template writes two lines: Disallow: /wp-admin/ and Allow: /wp-admin/admin-ajax.php. The admin area closes to crawlers, while admin-ajax.php, which many themes and plugins call on the front end, stays open. Because the Allow line is longer, Google applies Allow to that file; you can confirm it yourself in the tester.
The WooCommerce template adds the cart, checkout and account pages. On an English install, those usually live at /cart/, /checkout/ and /my-account/, and the template writes exactly those. If your store uses other slugs, replace the rows with your own paths. In e-commerce, though, most wasted crawling comes from filter and sort parameters. For example, a Disallow: /*?orderby= line closes the sorted duplicates of your category pages.
One warning from me: do not block /wp-content/ or /wp-includes/. Those folders hold theme files, CSS and JavaScript, and without them Google renders a broken page. In e-commerce consulting projects, my team and I plan category and filter structure together with robots.txt for exactly this reason.
Should you block AI bots?
Before you decide, separate two kinds of bots. Training bots (GPTBot, ClaudeBot, CCBot) collect your content for model training or a data archive. Search bots (OAI-SearchBot, PerplexityBot) crawl so that AI powered search answers can cite your site. Google-Extended is a special case: it is not a separate crawler but a control token for Gemini training and grounding, and it does not affect your place in Google Search.
My recommendation is simple. If you do not want your content in training data, block the training bots, but leave the search bots open. Otherwise your brand loses visibility on surfaces like ChatGPT search. OpenAI itself says that blocking GPTBot does not change ChatGPT search, which looks at the OAI-SearchBot rule instead.
Whatever you choose, keep two points in mind:
- robots.txt only works going forward; it cannot pull back data that bots collected earlier.
- Some fetchers that act on a user request, such as Perplexity-User, generally ignore robots.txt.
I cover how to grow that visibility in my article on how your brand shows up in ChatGPT and Gemini.
How do you test robots.txt after uploading it?
After the upload, I run three checks:
- Open it in a browser. Load https://yoursite.com/robots.txt in a private window. The file should show as plain text and return a 200 status. You can trace redirect chains with the redirect checker.
- Check the Search Console robots.txt report. For the top 20 hosts on your site, it lists the files Google found, the last crawl time and any parsing errors. If you removed an important block, you can request a recrawl from the same report.
- Test key URLs. Type representative URLs such as the home page, a category, a product and a blog post into this tool's tester and confirm that each gives the result you expect.
Google generally caches the file for up to 24 hours, so a change may not show its effect right away. If Search Console is new to you, start with my Google Search Console guide.
How do robots.txt, noindex and sitemaps work together?
These three do different jobs and complete each other. robots.txt controls crawling, noindex controls indexing and the sitemap helps discovery. Most problems come from mixing them up.
- noindex needs crawling. If you want noindex to drop a page from the index, do not block that page in robots.txt; Google only sees the tag when it crawls the page. You can prepare the tag with the meta tag generator.
- List only indexable URLs in the sitemap. Putting a URL you block in robots.txt into your sitemap sends Google a mixed signal.
- The Sitemap line needs a full URL. Google does not try http versus https or www versus non www variants; it reads the address as written.
If you have no sitemap yet, build one with the XML sitemap generator and add its URL to the Sitemap field in this tool. Then submit the sitemap in Search Console too; the robots.txt line helps discovery, while Search Console gives you the reporting.
A robots.txt checklist for launches and migrations
The most expensive robots.txt mistake I keep seeing is always the same one. The Disallow: / line from a staging site travels to production together with the file at launch. That single line stops crawling of the entire site, and organic traffic erodes until someone notices. This tool flags the line as critical, but a launch day checklist is the safer habit.
- Before you go live, open robots.txt and search for a Disallow: / line.
- Protect staging with a password instead of robots.txt; then even a copied file carries no risk.
- If the domain or protocol changed, for example from http to https, make sure the new host serves its own robots.txt. Rules only apply to the host, protocol and port where the file lives.
- Update the addresses in your Sitemap lines to the new domain.
For the whole move, use my website migration SEO checklist.
Common robots.txt mistakes
- ✕MistakeBlocking CSS, JavaScript or the /wp-includes/ folder.✓Do this insteadLeave resource files open, because Google renders pages like a browser and cannot see the layout without them.
- ✕MistakeCarrying the staging Disallow: / line to the live site.✓Do this insteadAdd robots.txt to your launch checklist and protect staging with a password.
- ✕MistakeUsing Disallow to remove a page from the index.✓Do this insteadUse noindex and keep the page crawlable; Google cannot drop a page for noindex without seeing the tag.
- ✕MistakeTrying to hide private pages with robots.txt.✓Do this insteadAnyone can read the file, and it reveals the paths you list; protect sensitive areas with a password or on the server.
- ✕MistakeForgetting the trailing slash on a directory.✓Do this insteadDisallow: /shop also blocks /shopping-guide/; if you mean only the directory, write /shop/.
- ✕MistakeWriting a relative path in the Sitemap line.✓Do this insteadWrite the full URL, for example https://yoursite.com/sitemap.xml; Google does not complete relative paths.
Frequently Asked Questions
Robots.txt done, what about the rest of your technical SEO?
Crawl management is only the first step; rankings are decided by site speed, content and authority. Get a quote for SEO consulting: share your site address and goal in the quote form, and we will send a written scope and quote.
