Robots.txt Generator & Tester
Build a clean robots.txt, or paste yours in and check it for mistakes — including testing which crawler can fetch which URL.
robots.txt is the first file most crawlers fetch from a site: a plain-text list of fetching rules, grouped by user-agent. It is a convention, not a security mechanism — anything you truly need to keep private must be behind a login, because a Disallow line actually advertises that the path exists.
How to use
- Start from a preset, or type a user-agent and the paths it may not fetch — the file builds itself as you type.
- Copy or download the result and place it at the root of your domain: yourdomain.com/robots.txt must return this exact text.
- Before deploying a hand-edited file, paste it into the tester below: it flags malformed lines, rules placed before any User-agent, and unknown directives.
- Use the tester’s verdict box to spot-check the cases you care about — your staging path as Googlebot, your whole site as an AI crawler — because the longest matching rule wins, and one broad Disallow can quietly override everything under it.
About this tool
The rules have two subtleties that cause most real-world mistakes. Groups are chosen by the most specific matching user-agent, so a Googlebot group replaces the * group entirely instead of adding to it. And when several rules match a URL, the longest path wins, with Allow breaking ties — which is how a single Allow for one file can reopen an otherwise closed folder. The tester on this page implements both rules, so the verdicts match how major crawlers actually decide.
Worked examples
The * group is not a fallback for named bots: A file with “User-agent: * / Disallow: /tmp/” plus “User-agent: Googlebot / Disallow: /private/” does not block Googlebot from /tmp/ — once a Googlebot group exists, Googlebot reads only that group and may fetch /tmp/ freely. Paste it into the tester and ask as Googlebot for /tmp/x: the verdict is ALLOWED. Repeat the /tmp/ rule inside the Googlebot group if you mean it for Google too.
One Allow reopens a folder: “Disallow: /admin/” plus “Allow: /admin/public/” blocks the admin area except the public subfolder, because /admin/public/ is the longer match for anything under it. That is exactly how WordPress sites keep admin-ajax reachable: Disallow /wp-admin/ with Allow /wp-admin/admin-ajax.php.
Quick reference
| Directive | Meaning | Notes |
|---|---|---|
| User-agent | Starts a group for one crawler (or * for all) | A crawler follows only its most specific group |
| Disallow | Paths this group may not fetch | Empty value = nothing is disallowed |
| Allow | Exception inside a Disallow | Longest match wins; Allow wins ties |
| Sitemap | Full URL of your sitemap | Not group-specific; may repeat |
| Crawl-delay | Seconds between requests | Honored by Bing/Yandex; Google ignores it |
Frequently asked questions
Does robots.txt keep a page out of Google search results?
Not reliably. It stops fetching, not indexing: if other sites link to a blocked URL, Google can still list it (with no description) because it knows the URL exists. To remove a page from results, use a noindex tag or a password — robots.txt alone is the wrong tool.
Can robots.txt protect private files?
No. It is a public file that anyone can read, and a Disallow line names the very paths you are hiding. Crawlers that ignore instructions — scrapers, malware scanners — will read the file and go straight for those paths. Private content needs authentication.
Do AI crawlers respect robots.txt?
The named ones from major labs (GPTBot, ClaudeBot, CCBot) state that they do, and blocking them is a User-agent group with Disallow: / each. Unnamed scrapers may not identify themselves honestly, so treat robots.txt as a signal of your preference, not enforcement.
Is anything uploaded when I test my file here?
No. Parsing, checking and the URL verdict all run in JavaScript in your browser — paste a staging or internal robots.txt without it leaving your device.
More free tools
Meta Tags Preview
Preview how your page looks on Google, X and Facebook — and grab the meta tags that make it look right.
QR Code Generator
Turn any link or text into a scannable QR code in seconds.
Word Counter
Count words, characters, sentences and reading time as you type.
Percentage Calculator
Work out percentages, increases and discounts in one click.
Age Calculator
Find your exact age in years, months and days.
BMI Calculator
Check your Body Mass Index in seconds.