Write robots.txt rules, then check exactly what they block
This robots.txt generator builds the file from groups of user agents and Allow/Disallow rules, adds your sitemap, and applies ready-made setups such as blocking AI training crawlers or a standard WordPress file in one click. The tester then answers the question that matters: is this URL allowed for this crawler, and which line decides it? It follows RFC 9309 and Google's documented behaviour — the most specific user-agent group, the longest matching path, Allow winning ties, and the * and $ wildcards — and flags mistakes as you type. Everything runs in your browser.
How to create and test a robots.txt
- 1Start from a preset — Allow all, Block AI crawlers, WordPress, Block SEO tools or Block all — or edit the groups directly.
- 2In each group, list the user agents it applies to (
*means every crawler) and add Disallow and Allow paths such as/admin/or/*.pdf$. - 3Add the absolute URL of your XML sitemap, check the warnings under the output, then copy or download
robots.txt. - 4Switch to Test, enter a crawler name and one URL per line, and see each verdict with the rule and group that decided it. Paste any live robots.txt there to audit it.
- 5Upload the file to the root of your host, so it is served at
https://yourdomain.com/robots.txt.
How robots.txt works
Before crawling a site, a well-behaved crawler fetches /robots.txt from the root of the host and looks for rules addressed to it. The file is a list of groups: one or more User-agent lines followed by Allow and Disallow rules. Sitemap lines stand outside the groups and apply to everyone.
Each protocol, host and port has its own robots.txt: rules at https://example.com/robots.txt do not cover https://shop.example.com or http://example.com. A missing file (404) means everything may be crawled; a server error makes Google pause crawling the site.
The matching rules, precisely
- Choosing the group
- A crawler obeys only the group whose user-agent most specifically names it, matched case-insensitively — Googlebot-Image uses a Googlebot-Image group if there is one, otherwise the Googlebot group, otherwise *. Groups naming the same agent are merged. Rules in the * group do not add to a named group.
- Longest match wins
- Within the group, the rule whose path matches the most characters of the URL path decides. Disallow: /shop with Allow: /shop/sale allows /shop/sale/shoes.
- Ties go to Allow
- If an Allow and a Disallow rule match with the same length, the URL is allowed.
- * and $
- * matches any sequence of characters, and $ at the end anchors the pattern to the end of the URL: Disallow: /*.pdf$ blocks /files/report.pdf but not /report.pdf?download=1.
- Paths are prefixes
- Disallow: /admin blocks /admin, /admin/ and /administrator. Add a trailing slash to block only the folder. Rules match the path and query string, case-sensitively.
- Empty Disallow
- Disallow: with no path allows everything; it is the conventional way to write a group that blocks nothing.
Blocking AI crawlers
AI companies publish the user-agent tokens of their crawlers so site owners can opt out. The Block AI crawlers preset disallows GPTBot, ChatGPT-User and OAI-SearchBot (OpenAI), ClaudeBot, Claude-User, Claude-SearchBot and anthropic-ai (Anthropic), Google-Extended (Gemini training), Applebot-Extended, CCBot (Common Crawl), PerplexityBot, Bytespider (ByteDance), meta-externalagent (Meta), Amazonbot and others, while leaving regular search engines free to crawl.
Google-Extended and Applebot-Extended are control tokens rather than separate crawlers: blocking them keeps your content out of Gemini and Apple Intelligence training without affecting Google Search or Applebot's search crawling. Note that the search-related bots such as OAI-SearchBot and Claude-SearchBot power citations in AI search answers; drop them from the list if you want to appear there. Compliance is voluntary, so use server or firewall rules for crawlers that ignore robots.txt.
What robots.txt cannot do
robots.txt controls crawling, not indexing. A disallowed URL can still appear in Google's results — without a description — if other pages link to it. To keep a page out of search results, allow crawling and add a noindex robots meta tag or X-Robots-Tag header; Google ignores Noindex: lines in robots.txt since 2019. Generate the tag with the meta tag generator.
Do not use robots.txt to hide private areas: the file is public and lists exactly the paths you would rather people did not find. Protect them with authentication. And do not block CSS and JavaScript files that pages need to render, or Google cannot see the page as visitors do.
Crawl-delay, Host and other directives
Crawl-delay asks a crawler to wait a number of seconds between requests. Bing and Yandex honour it; Google ignores it and adjusts its crawl rate automatically from your server's responses. Host was a Yandex-only directive for the preferred mirror and is obsolete. Google reads at most 500 KiB of a robots.txt file.
Point crawlers to your XML sitemap with one or more Sitemap: lines using absolute URLs; they may sit anywhere in the file. Build the sitemap itself with the sitemap generator.
Frequently asked questions
- Where do I put robots.txt?
- In the root of the host, so it is served at https://example.com/robots.txt. A robots.txt in a subfolder is ignored, and each subdomain needs its own.
- How do I block all crawlers?
- Use User-agent: * followed by Disallow: /. Crawlers that obey robots.txt will stop fetching pages, though URLs already known can stay in search results for a while.
- How do I block GPTBot and other AI crawlers?
- Add a group listing their user agents — GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot and so on — with Disallow: /. The Block AI crawlers preset writes this group for you and keeps search engines allowed.
- Does Disallow remove a page from Google?
- No. It stops crawling, but the URL can still be indexed from links. Use a noindex meta tag on a crawlable page, or remove it with Search Console, to keep it out of results.
- Which rule wins when Allow and Disallow both match?
- The one with the longer path. If both have the same length, Allow wins. The tester shows which line decided each URL.
- Is robots.txt case-sensitive?
- Directive names and user-agent values are not, but paths are: Disallow: /Admin does not block /admin. The file name itself must be lowercase robots.txt.
- Does Google support Crawl-delay?
- No. Google ignores Crawl-delay and sets its crawl rate from how your server responds. Bing and Yandex do honour it.
- Is my robots.txt uploaded anywhere?
- No. Building, parsing and testing happen in your browser.