Robots.txt Tester

Test whether a URL is allowed by your robots.txt rules.

  • 100% free
  • Private in browser
  • No signup
  • No watermarks

Paste robots.txt and enter a path to test.

Rate this tool

Be the first to rate — helps others and improves this page.

About Robots.txt Tester

One stray Disallow line in robots.txt can hide important pages from search engines. Robots.txt Tester on TheToolss lets you paste your robots.txt, enter any path and user-agent, and see whether it is allowed or blocked — plus the exact rule and group that decided it. It follows the standard longest-match logic (with Allow winning ties) and supports * and $ wildcards. Test changes safely before you deploy them. It runs entirely in your browser — free, private, and no signup.

How to use Robots.txt Tester

  1. Paste your robots.txt content.
  2. Enter the URL or path and a user-agent.
  3. See whether it is allowed and which rule matched.

Key features

  • Longest-match rule logic (Allow wins ties)
  • Supports * and $ wildcards
  • Per user-agent group selection
  • Shows the exact matched rule
  • Private in-browser testing

Frequently asked questions

Is the Robots.txt Tester free?
Yes. Test robots.txt rules free on TheToolss with no signup.
How does rule matching work?
Search engines apply the most specific group for the user-agent, then the rule with the longest matching path; if an Allow and Disallow tie, Allow wins.
Does it support wildcards?
Yes. * matches any sequence and $ anchors the end of the URL, matching how Google interprets robots.txt.
Does blocking in robots.txt remove a page from Google?
No. It only prevents crawling. To remove a page from the index, use a noindex tag or removal request instead.
Is my robots.txt uploaded?
No. Testing happens locally in your browser.
Robots.txt Tester — Check if a URL is Allowed | TheToolss