Free tool · Technical SEO
Free robots.txt generator
To write a robots.txt, group your rules under a User-agent line, list the paths to disallow, and add your sitemap as a full URL. Build one below from a preset or your own rules, then read it back through the same parser we run on live sites to see what each crawler will actually be allowed to do.
Free. Building and validating a file needs no account; the live diff against your deployed file needs a free sign-in.
The right default for almost every live site. An empty or permissive robots.txt is not a missed optimisation; blocking is the exception that needs a reason.
Your robots.txt
User-agent: * Allow: /
Read back through the parser
Parses cleanly into 1 group. Nothing here would be ignored by a crawler.
AEO depth layer
See the live robots.txt diff
The live robots.txt diff is computed from your own page and costs nothing to run. We ask for a free account because the expensive tools on this shelf stay free, and because it lets us tell you when your result changes. No card, no trial clock.
- The robots.txt your server is returning right now, next to the one you just built, so you can see what you are about to replace.
- A per-bot verdict on the live file for GPTBot, ClaudeBot, PerplexityBot and the search crawlers, using each vendor's own token.
- The catch this exists for: a staging block that was never lifted, still quietly keeping a live site out of search.
Signing in saves the details you entered above so we can follow up about your results. See our privacy policy for what we keep and why. The Robots.txt Generator result you already have stays free either way. Privacy policy.
The presets, and the tradeoff each one makes
Every preset states what it costs you as well as what it does. A robots.txt that blocks more is not a robots.txt that works better.
| Preset | Rules | What it means |
|---|---|---|
| Allow everything | Allow: / | The right default for almost every live site. An empty or permissive robots.txt is not a missed optimisation; blocking is the exception that needs a reason. |
| Block the whole site (staging) | Disallow: / | For a staging or preview host you do not want indexed. Correct there and catastrophic on a production domain, which is why the tool warns when you generate it. |
| Ecommerce | Disallow: /cart · Disallow: /checkout · Disallow: /account · Disallow: /search · Disallow: /*?sort= · Disallow: /*?filter= · Allow: / | Keeps crawlers out of carts, checkouts, account pages and faceted search URLs that generate near-infinite duplicates, while leaving products and categories open. |
| WordPress | Disallow: /wp-admin/ · Allow: /wp-admin/admin-ajax.php · Allow: / | Blocks wp-admin while leaving admin-ajax.php reachable, which plugins need, and leaves uploads open so images can be indexed. |
Two rules this tool applies that are our judgement rather than anyone's published limit: a Crawl-delay is only emitted between 1 and 60 seconds, because nothing honours a larger value and it reads as a mistake, and a Sitemap line must be an absolute http or https URL, because a relative one is ignored by every crawler.
How this compares to a typical free generator
Where the usual generators are equal, the table says so.
| Capability | This tool | Typical free generator |
|---|---|---|
| Presets and per-agent groups | Yes | Yes |
| Sitemap and crawl-delay lines | Yes | Yes |
| Output parsed back with a real RFC 9309 parser | Yes | No |
| Warns that the file blocks the whole site | Yes | Rarely |
| Reports a directive written before any User-agent | Yes | No |
| Refuses to emit a relative sitemap line | Yes | Emits it anyway |
| Rejects a nonsense crawl-delay | Yes | Emits it anyway |
| Strips injection attempts from pasted values | Yes | No |
| AI-crawler tokens named with each vendor's definition | Yes | Generic list at best |
| Diff against your live file | Yes, with a free account | No |
What is in the live robots.txt diff
Writing a correct file is the easy half. The half that goes wrong is what is already deployed, because nobody reads a robots.txt again after the day they wrote it.
- The file your server returns right now, shown beside the one you just built, so you can see exactly what you are about to replace.
- A per-bot verdict on the live file using each vendor's own token, separating a search block from a training opt-out. Only one of those removes you from AI answers.
- Validation of the live file, including directives that sit before any User-agent line and groups with no rules at all, both of which crawlers ignore in silence.
The single most common finding is a staging block that was never lifted. Once the file is right, the AI crawlability checker reports each AI bot against your live rules, the llms.txt generator builds the file that tells the engines what to cite, and the AI visibility checker answers the question being readable only makes possible: do they recommend you.
How to write a robots.txt file
Step 1
Start from the preset closest to your site
Allow everything is the right default for almost any live site. Blocking is the exception that needs a reason, not a missed optimisation.
Step 2
Read the parser output, not just the file
Under the generated file is what our parser makes of it. That is the part worth your attention: a file that looks right and parses wrong is the failure this tool exists to catch.
Step 3
Add your sitemap as a full URL
A Sitemap line must be absolute. A relative path is dropped here rather than emitted, because a crawler would ignore it and you would never know.
Step 4
Diff it against what is live before you replace anything
The live diff reads the file your server returns right now and gives a per-bot verdict on it. Then check whether the engines actually recommend you with the AI visibility checker.
When you outgrow this tool
A correct robots.txt is the floor: it only decides whether the engines may read you. Answer Radar tells you whether they then recommend you, and which competitor they name instead.
The generator is free and complete on its own. Linkeddit is what you use when being readable stops being the question and you need to know whether the engines actually recommend you, every week, without checking by hand.
FAQ
Robots.txt Generator questions
What the file does, the one thing it famously does not do, and what a generated file cannot tell you.
What should a robots.txt file contain?
For most live sites, as little as possible. A single group allowing everything is a perfectly good robots.txt, and an empty file means the same thing. The file's only job is to keep crawlers out of places that waste their time or yours: carts, checkouts, admin screens, and faceted search URLs that generate near-infinite duplicates. It is not a ranking lever and adding rules does not improve anything.
Does robots.txt stop a page being indexed?
No, and this is the most expensive misconception about the file. Disallow stops a crawler fetching the page; it does not remove the URL from search results. Google can still list a blocked URL it has learned about from links elsewhere, showing no description. If you want a page out of the index you need a noindex meta tag or header, which requires the crawler to be allowed to fetch the page and read it. Blocking in robots.txt actively prevents that.
Does robots.txt block AI crawlers?
Only the ones that choose to honour it, and only if you name their token. GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others each read a specific User-agent string, so a rule aimed at one does nothing to the others. There is also a distinction worth understanding before you write anything: a training opt-out does not remove you from AI answers, while a search block does. Our AI crawlability checker reports each token separately with each vendor's own definition.
Where does the robots.txt file go?
At the root of the host, exactly at /robots.txt, served as plain text. It applies to that host and port only, so a subdomain needs its own file and http and https are technically separate. A file at any other path has no effect at all, which is a common and silent mistake.
Do the Allow and Disallow lines need to be in a particular order?
No. RFC 9309 evaluates by longest match, not by position, and when two rules match with equal length the Allow wins. So a Disallow of /admin with an Allow of /admin/public leaves the public path reachable regardless of which line comes first. This tool writes Disallow lines before Allow lines only because that is how people expect to read the file.
What is in the layer that needs an account?
The live diff: it fetches the robots.txt your server is returning right now, shows it beside the file you just built, validates the live one, and gives a per-bot verdict on it for the AI and search crawlers. It costs nothing to run and no model is involved. The account exists because the tools on this shelf that do spend money stay free, and because the thing this catches most often is a staging block nobody remembered to lift.
How is this different from other free robots.txt generators?
Most of them emit a file and stop. This one parses its own output with the same code we use to audit live sites and shows you the result, so a file that reads correctly but parses wrong cannot leave here silently. It also refuses to emit things a crawler would ignore, like a relative sitemap or a nonsense crawl-delay, rather than including them to look thorough.
More free tools
Every tool on the shelf is free, with no card and no trial. Most need no account at all. See all free tools.
Reads your og:image file and reports its real dimensions, rather than trusting the width you declared in a meta tag.
Build JSON-LD for ten schema types with a hint on every field, then check it against the markup already on your page.
Build the JSON-LD properties that make a page attributable rather than eligible for a rich result, with the reason each one matters.
Score a page on whether an assistant could actually answer with it, across four pillars worth a hundred points.