Robots.txt Tester

Robots.txt Tester fetches a domain's live robots.txt file, parses every user-agent block and directive, and lets you test whether a specific URL path is allowed or disallowed for a given crawler. Whether you’re auditing your own site or a client’s, this page explains the robots.txt tester concept end-to-end and, gives you a free, working free robots.txt tester below.

How it works

Enter a domain (and optionally a path and a user-agent). We fetch {domain}/robots.txt directly, parse the Allow/Disallow/Sitemap directives per user-agent group, and evaluate the most specific matching rule against your path.

Try it now

What we check

• Whether robots.txt exists and returns HTTP 200
• Directive syntax (User-agent, Allow, Disallow, Sitemap, Crawl-delay)
• Which rule wins for a given user-agent and path
• Referenced sitemap URLs

How to fix common Robots.txt Tester issues

1. Remove overly broad Disallow: / rules left over from staging.
2. Make sure the sitemap directive points to a real, current sitemap.
3. Use specific Disallow paths instead of blocking entire directories that contain indexable content.
4. Test every important template path (blog post, product, category) before deploying changes.

FAQs