Tools
Robots.txt Generator
Build a custom robots.txt file for your site in seconds: ready-made templates, a line-by-line rule editor, sitemap support and one-click copy. Free, no sign-up.
Save the output as a file named robots.txt and upload it to the root directory of your site (e.g. public_html). The file must be reachable at https://yoursite.com/robots.txt; robots.txt files placed in subdirectories are ignored. Each subdomain uses its own file.
What Robots.txt Does and What It Cannot Do
robots.txt is a plain text file that tells search engine bots which parts of your site they may crawl. Used correctly, it manages your crawl budget: it keeps bots away from low-value URLs like carts, filters and admin pages, so they visit your important pages more often. But it has one critical limit: robots.txt does not remove a page from Google's index. A disallowed URL that receives links from other pages can keep appearing in search results without a description; removal is the job of the noindex tag or Search Console tools.
The most common mistake is blocking CSS and JavaScript files. Google renders your page like a browser; if it cannot access styles and scripts, it sees a broken page and may misjudge mobile friendliness and page experience. The second classic mistake is a leftover Disallow: / line from development that goes live; that single line stops crawling of the entire site. Also remember that paths are case sensitive: /Blog/ and /blog/ are different rules.
The Sitemap: line is the most practical way to announce your sitemap to bots. It must use an absolute URL (starting with https://) and multiple lines are allowed. The line is not mandatory, but it speeds up discovery, especially on new sites. Submitting your sitemap in Google Search Console as well is still the safest route; this tool automatically writes every sitemap address you add to the end of the output.
Robots.txt done, what about the rest of your technical SEO?
Crawl management is only the first step; rankings are decided by site speed, content and authority. Let's look at your site together in a free 15-minute call.