Robots.txt Wildcards: How to Write Rules With * and $ Correctly

In this guide, Talha Aslan and team walk through the wildcard questions we see most often in technical audits: what each character matches, which rule wins, and how to fix broken patterns. If you prefer to start from a clean template, keep our free robots.txt generator open while you read. That way, you can copy each example straight into your own file. At the end, you will also find a ready template and a testing routine.

New to the file itself? Start with our guide on what robots.txt is and how to create it. We do not repeat the basics here. Instead, we focus only on pattern matching.

What exactly does the asterisk (*) match?

The asterisk means "anything can go here." In other words, it can match zero characters or hundreds of them. For example, Disallow: /*.pdf covers every URL that contains ".pdf" anywhere in the path. So think of it as a short tool with a wide reach.

Google documents one detail that surprises many site owners: a trailing asterisk does nothing. The rule /fish* matches the same URLs as /fish, because robots.txt paths already work as prefix matches. In short, a trailing asterisk does no harm, but it adds nothing either.

• Disallow: /*?: blocks every URL that contains a question mark, which means every URL with parameters.
• A narrower case is Disallow: /product/*/reviews: it blocks any path under /product/ that contains "/reviews".
• A risky case is Disallow: /*cart: it blocks any URL that contains "cart" anywhere, including /cartoon-guide.

The last example shows how wide the asterisk can reach. For instance, a blog post with "cart" in its slug would drop out of crawling too. That breadth is both the benefit and the risk, and we return to it in the mistakes section below.

How does the dollar sign ($) limit a robots.txt rule?

The dollar sign anchors the match to the end of the URL. For example, Disallow: /*.pdf$ blocks only URLs that end in ".pdf". On the other hand, a URL such as /report.pdf?download=1 does not match. The reason is simple: that URL ends with a parameter, not with ".pdf".

Google's own example follows the same logic. The rule /*.php$ matches /folder/filename.php, but it does not match /filename.php?parameters or /filename.php/. Put simply, the dollar sign means "stop exactly here." So if you also want to block parameter versions, leave the dollar sign out or add a second line.

• File type case: Disallow: /*.xls$ blocks only files that end in .xls.
• Homepage case: Allow: /$ covers only the root URL, not the pages below it.
• Exact page case: Disallow: /search$ blocks only /search itself, not /search?q=shoes.

Which robots.txt rule wins when several rules match?

When more than one rule matches a URL, Google applies the most specific one. It measures specificity by the length of the rule path. Therefore, the longer path beats the shorter one. For example, with Allow: /blog/private/ and Disallow: /blog/, pages under /blog/private/ stay crawlable.

The same length logic applies to rules with wildcards. In Google's documented example, the URL /page.htm matches both allow: /page and disallow: /*.htm. The disallow rule is longer, so it wins. In other words, an asterisk does not make a rule weaker or stronger by itself. Length decides.

This point confuses many people, because they assume Disallow always comes first. So what happens when two matching rules have the same length? In that case, Google uses the least restrictive rule, which is the Allow rule. For example, if you write both Allow: /folder and Disallow: /folder, the /folder path stays crawlable.

How do you resolve an Allow and Disallow conflict?

The safest fix is to always write the exception as a longer, more specific path. That way, the outcome depends on a clear length difference, not on line order. Also, line order does not decide anything for Google. Moving a rule to the top or the bottom of the file changes nothing.

1. Start with the general block, for example Disallow: /account/.
2. Next, add the path that must stay open as a longer Allow rule, for example Allow: /account/help/.
3. Finally, test both paths with sample URLs and note which rule wins.

Example scenario: a store wants to block every /*? URL but keep pagination open. The rule Allow: /*?page= is longer than Disallow: /*?, so parameter URLs that start with ?page= stay open. However, a URL like ?color=red&page=2 does not match that Allow rule, because page is the second parameter.

Are robots.txt paths case sensitive?

Yes, paths are case sensitive. According to Google, the rule /fish does not match /Fish.asp. So if your site has both /Products and /products, you need two separate rules. Otherwise, one of them stays open.

On the other hand, the bot name in the User-agent line is not case sensitive. That means User-agent: googlebot and User-agent: Googlebot target the same crawler. Field names such as Disallow and Allow are not case sensitive either. Still, we recommend a capital first letter for readability.

• Path values are case sensitive, so /PDF/ and /pdf/ are different folders.
• Bot names are not case sensitive, so bingbot and Bingbot are the same.
• File extensions follow the path rule as well: /*.PDF$ does not cover lowercase .pdf files.

How do you block parameter URLs with a robots.txt wildcard?

Parameter URLs are one of the biggest drains on crawl activity. Filter, sort, and session parameters create hundreds of copies of the same content. That is why wildcard patterns help most here. We cover the canonical and filter side in our faceted navigation guide.

• Broadest option: Disallow: /*? blocks every URL with a question mark. It is also the riskiest choice.
• First position only: Disallow: /*?sort= blocks URLs where sort is the first parameter.
• Later positions: Disallow: /*&sort= applies when sort comes second or later.
• Session IDs: Disallow: /*sessionid= blocks session URLs in any parameter position.

Notice that targeting one parameter precisely usually takes two lines. One line uses ? and the other uses &. That is because the first parameter follows a question mark, while every later parameter follows an ampersand.

How do you block specific file types like .pdf or .docx?

To block a file type, combine the asterisk and the dollar sign. Then only URLs that end with that extension match. For example, Disallow: /*.pdf$ stops crawling of every PDF on the site. However, it leaves a blog post with "pdf" in its slug untouched.

• All PDFs: Disallow: /*.pdf$ blocks every URL that ends in .pdf.
• One folder only: Disallow: /documents/*.docx$ blocks .docx files inside /documents/.
• Watch out for Disallow: /*.pdf: without the dollar sign, it can also catch URLs such as /pdf-guide-download.

Keep one thing in mind, though. Robots.txt stops crawling; it does not guarantee removal from the index. A PDF with external links can still show up in results as a bare URL. If you want a file fully out of the index, the X-Robots-Tag header or noindex is the better tool. We compare them in our noindex vs nofollow vs robots.txt guide.

How do user-agent groups separate rules?

A robots.txt file is made of groups. Each group starts with one or more User-agent lines, and the Allow and Disallow lines below belong to that group. When a crawler reads the file, it follows only the single group that matches it most specifically. As a result, groups do not stack.

Google's documentation is clear on this. Bot-specific groups and the global (*) group do not combine. For example, if you write a separate group for Googlebot, Googlebot ignores the global group completely. Therefore, you must copy important global blocks into each bot-specific group as well.

However, if the file has several groups for the same bot, Google merges them into one. Still, we suggest keeping each bot in a single group for easier upkeep. For example, put the global group first and the bot-specific groups after it.

Which pattern targets one specific bot?

To target a specific bot, put the product token from that bot's documentation in the User-agent line. Then add the rules that should apply only to that bot. For example, you can open a separate group to keep the image crawler out of one folder.

• Image crawler: User-agent: Googlebot-Image with Disallow: /private-images/ keeps only Google's image crawler out of that folder. It does not affect regular Googlebot crawling.
• Global group: User-agent: * with Disallow: /admin/ closes the admin folder for every bot without its own group.
• Dedicated group: once you add a group that starts with User-agent: Googlebot, Googlebot reads that group instead of the global one.

Targeting AI crawlers such as GPTBot or ClaudeBot works the same way. We cover their names and the pitfalls in our AI crawlers guide. Always check current bot names on the provider's official page.

What are the most common robots.txt wildcard mistakes?

Most mistakes we see in the field come from a pattern that matches more or less than expected. The table below lists common broken patterns, why they fail, and what to write instead. For broader file errors, see our post on common robots.txt mistakes and fixes.

The first row matters most. According to Google, path values should start with a slash. So always open the rule with a slash, then add the asterisk.

Can a wildcard accidentally block CSS and JavaScript files?

Yes, a careless pattern can do that easily. For example, Disallow: /*? also blocks files like style.css?ver=3 that carry a version parameter. In that case, Google may not render the page correctly. As a result, it can misjudge the layout and mobile experience.

The fix is simple: add longer Allow rules next to the general block. For example, Allow: /*.css$ and Allow: /*.js$ open files without a version parameter. For files with parameters, add Allow: /*.css? and Allow: /*.js?. These lines are longer than Disallow: /*?, so they win.

If you run WordPress, the location of theme and plugin files matters too. We cover those details in our sibling guide on the best WordPress robots.txt setup.

How do you test a robots.txt rule before it goes live?

Testing a wildcard pattern before launch costs far less than explaining a traffic drop later. Our team follows the steps below on every robots.txt change. You can follow the same order.

1. List 5 to 10 sample URLs you want blocked and 5 to 10 URLs that must stay open.
2. Draft the rules in our robots.txt generator and check each user-agent group on its own.
3. For each sample URL, work out which rule matches and how long its path is; the longest wins, and Allow wins a tie.
4. Upload the file to the root of the host as UTF-8 plain text, because crawlers ignore a file in a subfolder.
5. Use the robots.txt report in Search Console to confirm that Google fetched the file without errors.
6. A few days later, review crawl stats and indexing reports; if you see an unexpected drop, roll the rule back.

Google publishes placement and encoding details in its guide to creating a robots.txt file. In addition, the full matching logic lives in Google's robots.txt specification.

Does rule order matter in a robots.txt file?

For Google, no. Google does not look at line order; it looks at length and restrictiveness. So placing an Allow rule above or below a Disallow rule changes nothing. That gives you freedom to organize the file for readability.

However, not every crawler behaves the same way. Some older or simpler crawlers may apply the first matching rule. That is why we suggest a practical habit: put exception Allow lines directly above the related Disallow line. That way, Google and order-based crawlers reach the same result.

How do robots.txt wildcard patterns affect crawl budget?

Well-written patterns cut the requests Googlebot spends on low-value parameter copies. As a result, important pages get crawled more steadily. The effect is clearest on stores that generate thousands of filter combinations.

On the other hand, an overly broad pattern can push important pages out of crawling. If you see an unexpected drop in crawl rate, the cause is sometimes a single asterisk. You can use the checklist in our post on why Googlebot crawls less to investigate.

• First, find the URL patterns that receive the most requests in Search Console crawl stats.
• Then confirm that these patterns are truly low-value copies.
• Finally, close only the confirmed patterns with narrow, specific rules.

Does the Sitemap line accept wildcards?

No. This question comes up often, because people who get used to wildcards want to use them on every line. According to Google, the asterisk works in every rule except sitemap. The Sitemap line needs a complete, valid URL, for example Sitemap: https://example.com/sitemap.xml. Also, this line is not tied to any user-agent group, so it can sit anywhere in the file.

If you have more than one sitemap, write a separate Sitemap line for each. If you need to build a sitemap from scratch, our XML sitemap generator cuts the job down to a few minutes.

What does a ready robots.txt wildcard template look like?

The example below shows a sensible starting point for a typical online store. We note what each line does. However, this is an example scenario, so adapt it to your own URL structure before you copy it.

• User-agent: *: every bot without its own group reads this group.
• Cart pages: Disallow: /cart/ keeps cart pages out of crawling.
• Member area: Disallow: /account/ closes customer account pages.
• Sorting: Disallow: /*?sort= and Disallow: /*&sort= block the sort parameter in any position.
• File type: Disallow: /*.pdf$ blocks files that end in .pdf.
• Assets: Allow: /*.css$ and Allow: /*.js$ keep style and script files open.
• Sitemap: Sitemap: https://example.com/sitemap.xml declares the full sitemap address.

In short, a good template is short, readable, and tested. A long, complex file makes debugging harder. In addition, pairing each line with a test URL saves time on later changes. For example, when you add a new filter, you only update the related line.

Is there a size limit for robots.txt?

Yes. Google enforces a 500 KiB limit on robots.txt and ignores content after that limit. So a file that tries to block every URL one by one with thousands of lines can exceed it. Then the rules at the end of the file never apply.

That is exactly why wildcards are valuable. For example, instead of listing hundreds of filter URLs, you write a single line such as Disallow: /*?filter=. As a result, the file stays short and easy to maintain. Still, check the current limit in the official documentation.

How does a trailing slash change a rule?

A slash looks small, but it changes matching a lot. In Google's example, the rule /fish/ matches /fish/salmon.htm but not /fish. That is because the rule requires the path to start with "/fish/".

On the other hand, if you drop the trailing slash, the rule reaches much further. For example, Disallow: /blog closes the /blog/ folder, the /blog.html file, and even /blogger-guide. Therefore, add the slash on purpose when you block a folder.

• To close the whole folder, write Disallow: /blog/.
• To also close pages that begin with the same letters, use Disallow: /blog without the slash.
• For one page only, anchor it with the dollar sign: Disallow: /blog$.

In short, after writing each rule, ask yourself which other URLs this prefix catches. That one question prevents most of the mistakes we see in the field.

How do wildcards handle encoded and non-ASCII URLs?

When crawlers compare URL paths, they generally rely on the percent-encoded form. So if your URLs contain letters such as é or ü, the path sent to the server is actually an encoded string like %C3%BC. That is why you need to think about both forms when you write a rule.

Google expects robots.txt to be UTF-8 plain text and may ignore characters outside UTF-8. As a result, a badly encoded line can quietly become invalid. For that reason, we recommend building your slug structure with plain Latin letters from the start, instead of putting special characters into rules.

• Before writing a rule, copy the path from the browser address bar and check its encoded form.
• For paths with special characters, add both encoded and plain versions to your test list.
• For new pages, set a slug rule without special characters, which also makes robots.txt upkeep easier.

Do robots.txt wildcard rules cover subdomains?

No. Each robots.txt file applies only to the host, protocol, and port where it lives. For example, example.com/robots.txt does not affect shop.example.com. So you prepare a separate file for every subdomain.

In addition, HTTP and HTTPS versions count as separate hosts. In practice, if your site redirects to one version, the main file belongs on that version. Meanwhile, a staging subdomain usually keeps a stricter file that blocks everything. However, make sure that strict file never moves to the live site at launch. It is one of the most expensive mistakes we know.

How do you calculate a wildcard match by hand?

Instead of guessing which rule catches a URL, you can do a small calculation. First, list every rule that matches the URL. Then count the path length of each rule in characters. The longest one wins, and Allow wins a tie.

1. Write down the URL, for example /products/shoes?color=black.
2. Find the matching rules: Disallow: /*? (3 characters) and Allow: /products/ (10 characters).
3. Compare lengths: the Allow rule is longer, so it wins and the URL stays crawlable.
4. Check the result against your intent; if it surprises you, add a longer Disallow rule such as Disallow: /products/*?.

This worked example shows why a wildcard alone is not enough. The general /*? block is short, so folder-level Allow rules beat it easily. Therefore, if you also want to close parameters inside allowed folders, write a longer, folder-specific Disallow rule.

How should a team manage robots.txt changes?

Robots.txt is a small text file, yet one line can change the visibility of a whole site. That is why we suggest you stop treating it as a file anyone can edit at random. On our team, every change gets a note and a date.

• Before each change, save a dated copy of the current file.
• Document who made the change and why in a short note.
• Next to each new wildcard line, add a comment that names the URL group it targets; lines that start with a hash (#) are comments.
• After every launch, theme update, or migration, check the file again.

For example, during a migration the staging file that blocks the whole site can slip into production. Then crawling drops fast, and the problem only shows up as lost traffic. Adding a robots.txt step to your launch checklist is one of the cheapest safeguards you can have. Also, comments help a teammate understand months later why a rule exists.

When is robots.txt not enough?

Robots.txt manages crawling only. Problems like index bloat, duplicate content, or mixed canonical signals usually need several tools working together. For example, parameter problems often call for robots.txt, canonical tags, and clean internal linking at the same time.

If your site is large, or you lost traffic after a robots.txt change, we review this file as part of a technical audit within our SEO consulting work. That said, most small and mid-sized sites can write their own file safely with the rules in this guide.

Core guide: What Is Digital Marketing? Its types, examples and how it is done, on one page with an interactive film that travels through the brain.