robots.txt tester & builder
Paste your rules and test a path per bot, or build a robots.txt from scratch. Uses the Google matching model — longest rule wins, Allow beats Disallow.
robots.txt
Rules are grouped by User-agent. Within the group that matches the bot, the rule with the longest path pattern wins; if an Allow and a Disallow tie, Allow wins.
Test a path
Build from a form
Serve this at https://yourdomain.com/robots.txt. robots.txt controls crawling, not indexing — to keep a page out of results use a noindex meta tag or header on a page that is still crawlable.
What robots.txt controls — and what it does not
robots.txt controls crawling, not indexing. A Disallow rule tells compliant bots not to fetch matching URLs, but a disallowed page can still be indexed from external links, just without a snippet. To keep a page out of search results, let it be crawled and add a noindex meta tag or X-Robots-Tag header. This tool tests whether a path is crawlable for a given bot against your rules and builds a robots.txt from a form. It uses the Google matching model: the bot picks the User-agent group that most specifically matches its name, then within that group the rule with the longest path pattern wins, and if an Allow and a Disallow tie on length, Allow wins.
Wildcards, groups and common mistakes
The wildcards * (any sequence) and $ (end of path) are supported by Google, Bing and Yandex. A frequent surprise: Disallow: /search blocks every path that starts with /search, including /search/help and /search?q=x — write Disallow: /search$ to block only the exact page, or add Allow: /search/help to keep one path open. Different bots need separate User-agent groups, and a bot uses only its single most specific group and ignores the rest, so repeat shared rules in each group. Crawl-delay is ignored by Google (set the rate in Search Console) but honoured by Bing and Yandex.