Create a clean, valid robots.txt file in a minute. Use presets for WordPress and Shopify, block or allow AI crawlers, add your sitemap and test any URL before you upload.
Free SEO tool · 2026
Create a clean, error free robots.txt in a minute. Control Google, Bing and AI crawlers, test any URL, then copy or download the file. Everything runs in your browser.
User-agent: *.
Custom selection. Pick a choice above to reset.
| Bot | Company | Used for | Rule |
|---|
Blocking Google-Extended does not change your normal Google rankings or AI Overviews. "User visits" bots open a page when a person asks the AI to, and may not always follow robots.txt.
These are added under User-agent: *.
A robots.txt file is a small text file at the root of your site, like yourbrand.com/robots.txt. Search engine and AI bots read it before they crawl, to see which parts of your site they may visit.
It controls crawling, not indexing. A blocked page can still show in Google if other sites link to it. To keep a page out of search results, use a noindex tag and leave the page crawlable.
Disallow: / from a staging site on the live site.No. Robots.txt only stops crawling. If other pages link to a blocked URL, Google can still list it without a description. To remove a page, add a noindex tag, password protect it, or delete it.
It depends on your goals. Training bots like GPTBot, ClaudeBot, Google-Extended and CCBot collect content to train AI models. Search bots like OAI-SearchBot, Claude-SearchBot and PerplexityBot decide whether you can be cited in AI answers. Many sites block training bots and allow search bots.
WordPress creates a virtual robots.txt for you. Edit it with Rank Math or Yoast, or upload a real robots.txt file to your root folder, which takes priority over the virtual one.
Disallow (in robots.txt) tells bots not to crawl a URL. Noindex (a meta tag or HTTP header) tells search engines not to show a page in results. Google must be able to crawl a page to see its noindex tag, so do not use both on the same page.
No. Google ignores Crawl-delay. Bing and Yandex respect it. If Google crawls too much, the crawl rate adjusts automatically based on how fast your server responds.
Google reads the first 500 KB of the file and ignores the rest. Most sites need far less than 1 KB.
Free tool by SpellMistake. Nothing you type is sent to a server.
A small mistake in robots.txt can block Google from your whole site. The generator avoids this by building rules from tested presets, checking for common problems and letting you test any URL before the file goes live. It also makes it easy to decide which AI crawlers, like GPTBot or Google-Extended, can use your content.
https://example.com/robots.txt.Important: robots.txt stops crawling, not indexing. To keep a page out of Google, use a noindex tag instead.
| Line | Example | What it does |
|---|---|---|
| User-agent | User-agent: * | Says which crawler the rules below are for. * means all crawlers. |
| Disallow | Disallow: /wp-admin/ | Blocks crawling of a path |
| Allow | Allow: /wp-admin/admin-ajax.php | Allows a path inside a blocked folder |
| Sitemap | Sitemap: https://example.com/sitemap.xml | Tells crawlers where your sitemap is |
| Crawl-delay | Crawl-delay: 10 | Asks bots to slow down. Bing and Yandex use it. Google ignores it. |
| * and $ | Disallow: /*.pdf$ | * matches any text, $ marks the end of the URL |
Each preset is a tested set of rules for that platform, like WordPress or Shopify.
In your browserRules are grouped by user-agent, including a separate group for each AI crawler you block.
In your browserThe tool flags risky rules, like blocking the whole site, blocking CSS and JS, or a missing sitemap.
In your browserThe tester applies Google's matching rules to show if a URL is allowed or blocked, and which line decided it.
In your browserClean, commented rules you can copy or download as robots.txt.
In your browserUser-agent group that matches them, or * if none does./Photos/ and /photos/ are different.* matches any characters and $ matches the end of the URL.User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / Sitemap: https://example.com/sitemap.xml
The file and the tester follow Google’s published rules. Other crawlers may read some lines differently. For example, Bing uses Crawl-delay and Google does not. AI companies can also add new crawler names, which we review every month.
WordPress, Shopify, allow all, block all and more.
Block AI training, block all AI, or allow every AI bot.
Add your sitemap line from your website URL in one click.
Warnings for risky rules before you upload.
See if any URL is allowed or blocked for any bot.
Paste your existing robots.txt and improve it.
Use a safe WordPress preset instead of guessing.
Stop bots from crawling cart, filter and search pages.
Choose which AI crawlers can use your content.
Build and test client files quickly and safely.
First release with presets, AI crawler controls, sitemap auto fill, health check and URL tester.
We run the tool against our test cases again and check RFC 9309, Google's robots.txt documentation and current AI crawler names for changes.
Tool page last reviewed on .
Tell us what you entered, what happened and what you expected. Our developer checks every report, fixes confirmed bugs as a high priority and tests the fix before it goes live.
This page follows our Editorial Policy and Methodology. If something here is wrong, our Corrections Policy explains how we fix it. Learn more about the SpellMistake team.
No ratings yet. Be the first to rate this tool.
How useful was the Robots.txt Generator for you?
Ratings come from real visitors. One rating per person. We never add, edit or remove ratings.
A robots.txt file is a plain text file at the root of your website. It tells search engine and AI crawlers which pages or folders they may crawl.
Yes. It is 100% free, with no sign-up and no limits.
Upload it to the root of your domain, so it opens at https://yourdomain.com/robots.txt. Each subdomain needs its own file.
No. It only stops crawling. Google can still index a blocked URL if other sites link to it. Use a noindex tag to keep a page out of search results.
Pick "Block AI training only" or "Block all AI" in the generator. It adds Disallow rules for bots like GPTBot, Google-Extended and ClaudeBot.
No. Google ignores Crawl-delay. Bing and Yandex follow it.
It blocks the whole site for that crawler. Only use it on purpose, for example to block one AI bot, never under User-agent: * on a live site.