Robots.txt Generator

Generate standards-oriented crawler directives without confusing crawl blocking with noindex.

User-agent

Disallow paths — one per line

Allow paths — one per line

Sitemap URLs — one per line

robots.txt controls crawling, not guaranteed de-indexingIf a page must be excluded from search results, use an appropriate noindex mechanism while still allowing the crawler to access that directive.

How to use this Robots.txt Generator

Build a robots.txt file from crawler groups, Allow/Disallow rules, and sitemap URLs. Enter your values above and the result is calculated directly in your browser. No account is required.

Fast answer, clear inputs

This tool is built around one job: give the answer quickly without forcing a signup or hiding the result. Related tools are linked nearby when the next calculation or conversion is useful.