Loading Techie Utils…

robots.txt Generator

Crawl rules, with the traps called out. Free, and runs entirely in your browser — nothing you enter is uploaded, stored or sent to a server.

About robots.txt Generator

robots.txt sits at the root of a site and tells crawlers where not to go. It is four lines of syntax with two large traps: it controls crawling rather than indexing, and blocking everything is one word away from allowing everything.

The generator writes the file for a chosen user agent, with allow and disallow paths, an optional crawl delay and — most usefully — the sitemap declaration, which is the first place a crawler looks for one.

Both traps are called out in the findings. Disallowing a page does not remove it from search: a page linked from elsewhere can still be indexed with no description, and the way to keep it out is a noindex tag on a page crawlers are *allowed* to fetch. And crawl-delay is ignored by Google entirely.

The score works the way it does everywhere in this category, with one heavy penalty: a file that blocks every crawler grades F, because on a live site that is the single most expensive mistake in SEO.

Where it helps

  • Writes a valid file for any user agent, with allow rules ordered before the disallows they carve out of.
  • Declares the sitemap, which is the single most useful line in the file.
  • Explains the crawl-versus-index trap plainly instead of assuming it is understood.
  • Says where the file has to live, which is the other thing people get wrong.

Limits

  • The file must be served from the root of the host, over the same protocol — a robots.txt in a subdirectory does nothing.
  • Wildcards and $ are honoured by the major engines but are not in the original standard, so an obscure crawler may ignore them.
  • It is public: anything you disallow is a list of paths you would rather people did not visit.
  • Malicious crawlers ignore the file entirely; it is a convention, not an enforcement mechanism.

Frequently asked questions

Will disallowing a page remove it from Google?

No. It stops the page being crawled, but it can still be indexed from links elsewhere, showing with no description. Use a noindex tag on a crawlable page instead.

Where does the file go?

At the root: https://example.com/robots.txt. One per host and protocol, and a subdomain needs its own.

Does Google honour crawl-delay?

No. Bing and Yandex do. For Google, adjust the crawl rate in Search Console if crawling is actually causing load.

How do I block one crawler but not others?

Give that crawler its own User-agent group with its own rules. A crawler obeys the most specific group that names it, and ignores the rest.

Related tools

More SEO & Meta tools