Robots.txt Tester & Generator
Test simple robots.txt rules against a URL path and generate a clean starter robots.txt file.
WHAT TO KNOW
How robots.txt rules affect crawling
Robots.txt provides instructions about which paths compliant crawlers may request. Test rules against the user agents and URLs that matter to your website.
Blocking crawling is not the same as removing a URL from search results. Robots.txt is not a security or access-control mechanism.
Test a robots.txt rule
This is a practical rule checker for common robots.txt patterns. Complex crawler-specific edge cases should be verified against the relevant crawler documentation.
Help us keep improving the Robots.txt Tester & Generator
Found the Robots.txt Tester & Generator useful? Give it a ★.
Your stars help us understand which free tools people value most. We use this feedback to decide which tools to maintain, improve and develop further.
You can support a tool once per visit, and every star contributes to its ongoing total.
Thanks for your input — it helps us focus our time on the tools that are most useful to you.
Star this toolFound an issue with this tool?
If something isn't working as expected, please let us know so we can investigate and improve it.
Report an issue