Robots.txt Tester and Generator

What robots.txt controls

robots.txt is a plain text file at the root of your site that tells crawlers which paths they may visit. It controls crawling, not indexing. A page blocked in robots.txt can still show up in search results without a description if other sites link to it. To keep a page out of search, use a noindex tag instead and leave the page crawlable.

Using the robots.txt tester

1. Enter the full URL of a page you want to test, not just the domain.
2. Pick the crawler to test as, for example Googlebot or GPTBot.
3. To try new rules before uploading them, open the option below the box and paste your draft file.
4. Run the test. You will see Allowed or Blocked, and the exact line that decided it is highlighted in the file.

Need a new file instead? Skip to the robots.txt generator further down this page.

How the robots.txt checker matches rules

The tool follows the matching rules Google publishes. First it finds the group of rules for the most specific matching user agent, so a Googlebot group wins over a group for every crawler. Inside that group, the rule with the longest matching path wins, and when an Allow and a Disallow rule are equally long, Allow wins. The wildcard * matches any characters and $ marks the end of a URL, so Disallow: /*.pdf$ blocks every PDF.

Common robots.txt mistakes

• Disallow: / under User-agent: * blocks the entire site. This often gets copied from a staging site by accident.
• Blocking CSS, JavaScript or theme folders stops Google from rendering your pages properly.
• Using robots.txt to hide private pages. The file is public, so it actually points people to those paths. Use passwords or noindex instead.
• Server errors on robots.txt. A 5xx error can make Google pause crawling the whole site.
• No Sitemap line. Adding one helps every crawler find your sitemap.

The robots.txt checker above flags most of these automatically whenever you test a URL.

Using the robots.txt generator

The generator below the tester builds a clean file in a few clicks. Choose whether to allow or block everything by default, list folders to block one per line, add exceptions, and paste your sitemap URL. Presets add the usual rules for WordPress and online stores, and an option adds rules for common AI training crawlers if you do not want your content used for that. Copy or download the result and upload it as robots.txt to the root of your domain, then test a few URLs with the tester above to confirm it works as expected.

After updating robots.txt, the sitemap and robots.txt scan confirms that your sitemap is listed and reachable.

Official reference: Google's introduction to robots.txt explains what the file can and cannot do, including why it is not a way to hide pages.