Check your robots.txt line by line against Google’s rules. Find typos and blocking mistakes, then test any URL against Googlebot, Bingbot or AI crawlers in seconds.
Robots.txt mistakes are easy to make and hard to spot. A missing colon, a rule outside a User-agent group or one extra slash can block important pages from Google. This tool checks your file against Google’s published rules and RFC 9309, explains each problem in plain words and shows the exact line that blocks a URL.
Disallow: / under User-agent: * blocks your whole site.User-agent line are ignored by Google.Noindex: in robots.txt does not work. Google stopped reading it in 2019./Blog/ is not /blog/.| Issue | Level | Why it matters |
|---|---|---|
| Whole site blocked | Error | Google cannot crawl any page |
| Line without a colon or directive | Error | Crawlers cannot read the line |
| Rule outside a User-agent group | Error | Google ignores it |
| CSS or JS blocked | Warning | Google cannot render your pages |
| Allow and Disallow for the same path | Warning | The result may not be what you expect |
| Noindex or Crawl-delay lines | Info | Google ignores them |
| File over 500 KB | Warning | Google only reads the first 500 KB |
| File starts with a BOM | Info | Some tools misread the first line |
Your live file is fetched from your domain, or the text you paste is used as it is.
Our server or your browserLines are split into directive and value. Comments and blank lines are handled.
In your browserRules are grouped under their User-agent lines, just like Google does.
In your browserEach line is checked against Google's rules and RFC 9309, and graded as error, warning or info.
In your browserThe tester picks the right group for the crawler and applies longest-match to show the deciding line.
In your browser* matches any characters, $ matches the end of the URL.The tool follows Google’s rules. Other search engines and AI bots may read a few lines differently. The final word on Google is the robots.txt report in Google Search Console.
Load the robots.txt from any domain in one click.
Problems are marked right on the line that has them.
Errors, warnings and info, so you fix the right things first.
Test any URL for Googlebot, Bingbot, GPTBot and more.
See the exact rule that allows or blocks a URL.
Every issue says what to change, without jargon.
Audit client files and find blocking mistakes fast.
Test robots.txt changes before a release.
Make sure Google can crawl your important pages.
Check that AI crawler blocks work as planned.
First release with live fetch, line by line checks, graded issues and a URL tester for any crawler.
We run the tool against our test cases again and check RFC 9309 and Google's robots.txt documentation for changes.
Tool page last reviewed on .
Tell us what you entered, what happened and what you expected. Our developer checks every report, fixes confirmed bugs as a high priority and tests the fix before it goes live.
This page follows our Editorial Policy and Methodology. If something here is wrong, our Corrections Policy explains how we fix it. Learn more about the SpellMistake team.
No ratings yet. Be the first to rate this tool.
How useful was the Robots.txt Validator for you?
Ratings come from real visitors. One rating per person. We never add, edit or remove ratings.
It checks your robots.txt file for errors and risky rules, and shows how Google reads it. A tester also tells you if a URL is allowed or blocked.
Yes. It is 100% free, with no sign-up and no limits.
Yes. Enter your domain and click "Fetch robots.txt" to load the live file, then run the check.
Enter the URL in the tester, choose Googlebot and click Test. The tool shows if it is allowed or blocked and which line decided it.
Google ignores rules outside a User-agent group, Noindex lines, Crawl-delay and anything after the first 500 KB of the file.
Not directly. But blocking important pages, CSS or JS can stop Google from crawling and understanding your site, which can hurt your search visibility.
Check it after every change, after a site move or redesign, and whenever pages drop out of Google.