Loading Techie Utils…

robots.txt Tester

Test whether URLs are allowed or blocked for Googlebot and other crawlers, and which rule decides. Free, and runs in your browser.

About robots.txt Tester

Tests whether a URL may be crawled under a robots.txt, for a chosen crawler, and shows the exact line that decides it. Paste the file and a list of URLs or paths, and each one is marked allowed or blocked.

It follows the standard (RFC 9309) the way Google does, including the rules that catch people out. A crawler obeys only the most specific group that names it — Googlebot-Image uses its own group if there is one, not Googlebot's, and a crawler with a group of its own ignores the * group entirely. Within the group the longest matching rule wins, and Allow beats Disallow on a tie, whatever order they are written in. * matches anything and $ anchors the end.

The file is checked as it is read: directives Google ignores (Crawl-delay, Noindex), misspelled ones, rules that belong to no group, paths that do not start with /, and a missing Sitemap line are all listed.

Where it helps

  • Chooses the group the way crawlers do — most specific match, groups merged, * only as a fallback — which is where most robots.txt mistakes hide.
  • Applies longest-match and Allow-on-tie precedence, wildcards and end anchors exactly as Google does.
  • Shows the line that decided each URL, so a surprising result can be traced and fixed.
  • Flags directives Google ignores, misspellings, orphaned rules and a missing sitemap.

Limits

  • Not every crawler follows the standard as closely as Google; some older bots ignore wildcards or Allow.
  • robots.txt controls crawling, not indexing: a blocked URL that other sites link to can still appear in results. Use noindex to keep a page out.
  • It tests the file you paste. To test a live site's file, copy it from /robots.txt first — or run the SEO Analyzer, which fetches it.

Frequently asked questions

Why is a URL allowed for Googlebot when the * group disallows it?

Because Googlebot has a group of its own. A crawler follows only the most specific group that names it and ignores the rest, including *. Repeat the rules that should apply to everyone inside the Googlebot group.

Does the order of Allow and Disallow lines matter?

Not to Google. The rule with the longest matching path wins, and when an Allow and a Disallow are the same length, Allow wins. Some older crawlers use the first match instead, so putting the more specific rule first keeps them in line too.

How do I stop AI crawlers from using my site?

Add a group for each one — GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, CCBot (Common Crawl), Google-Extended (Google AI training) — with Disallow: /. Test each here to confirm, and remember that robots.txt is a request that reputable crawlers honour, not an enforcement mechanism.

Will blocking a page in robots.txt remove it from Google?

No. It stops Google crawling the page, but a URL that is linked to can still be indexed and shown without a description. To remove a page, allow crawling and add a noindex meta tag or X-Robots-Tag header so Google can see it.

Related tools

  • robots.txt Generator — Crawl rules, with the traps called out
  • SEO Analyzer — Audit a live page's on-page SEO — indexability, title, headings, canonical, structured data and more — with a score
  • Sitemap Generator — An XML sitemap from a list of addresses
  • User-Agent Parser — Identify the browser, engine, operating system and device behind a User-Agent string
  • SERP Preview — Your page as a search result, measured in pixels
  • Meta Tag Generator — The head tags a page needs, with a search-result preview

More SEO & Meta tools