Free tool
Robots.txt Generator: Free, Nothing Uploaded
Build a robots.txt file or load your own, add rules for single bots, and test any path to see whether it is allowed. Warnings catch the common mistakes. Facts read from Google's documentation on 22 October 2026.
These are common starting points. Check the paths against your own site before you publish.
Everything runs in your browser. Nothing is uploaded.
How the generator works
Pick a starting point or build the file from scratch: who may crawl, which paths to block, which to allow inside a blocked path, your sitemap addresses, and rules for individual bots. The file on the right updates as you type, with warnings for the common mistakes. Copy it, or download it as robots.txt and upload it to the root of your site. If you already have a file, paste it into the box and load it into the form. The path tester then shows whether any path is allowed or blocked for any bot. Your last settings are saved on this device, and nothing you type leaves your browser.
A group in the file starts with a User-agent line, which names the crawler. A star means every crawler. Under it, Disallow blocks a path and Allow opens a path inside a blocked one. A Sitemap line gives the full address of your sitemap.
What a robots.txt file can and cannot do
| Question | Answer, per Google's documentation (read 22 October 2026) |
|---|---|
| What is it for? | Mainly to manage crawler traffic, so a site is not overloaded with requests |
| Does it keep a page out of Google? | No. A blocked page can still show up in results as a bare URL, without a description. Use a robots meta tag that tells search engines not to index the page, or password protection, to keep a page out |
| Is it a security measure? | No. Reputable crawlers follow it, but others may ignore it |
| Where does the file go? | It must be named robots.txt and sit at the root of the host it covers. A site has one file per host, saved as UTF-8 |
| Which rules does Google support? | user-agent, allow, disallow and sitemap. A star works in every rule except sitemap. Crawl-delay is not supported, so this tool does not add it |
| How big can it be? | Google reads up to 500 KiB and ignores the rest |
| How long is it cached? | Google generally caches the file for up to 24 hours, so a change may take a day to be noticed |
Sources: Google Search Central, introduction to robots.txt (updated 10 December 2025), how to create a robots.txt file (updated 21 November 2025) and how Google interprets the file (updated 31 August 2026).
How Google matches the rules
Each crawler follows only one group, the most specific group that matches its name, and the order of groups does not matter. Inside that group, the most specific rule wins, which means the longest matching path. If an Allow and a Disallow are equally specific, Google applies the less restrictive one, which is Allow. Paths are case-sensitive, so /fish does not match /Fish.asp, while field names are not. A star matches any run of characters, and a dollar sign marks the end of the address. The path tester above follows these rules.
What happens when the file is missing or broken
Google treats a 4xx response, except 429, as if there were no robots.txt, so it assumes no limits. A 5xx response pauses crawling for the first 12 hours while Google retries, and then it uses the last good copy for up to 30 days. This is why a site that breaks its robots.txt file can quietly lose crawling.
Blocking AI crawlers
The tool can add a block for crawlers run by AI companies. This only works for crawlers that choose to follow robots.txt, and the names change, so check each company's own documentation before you rely on a list. Blocking is a choice with a cost: a site that blocks a crawler may be left out of that company's answers.
Mistakes to avoid
- Writing
Disallow: /on a live site. It blocks everything, and it is often left in place when a staging copy goes live. - Blocking the folders that hold your CSS and scripts, which can stop crawlers from drawing the page the way visitors see it.
- Using robots.txt to hide pages. A robots meta tag that says not to index, or a login, is the right tool for that.
- Leaving out the full sitemap address. The sitemap line needs the whole URL, starting with https://.
Our own file is two lines of rules and a sitemap line: everyone is allowed, and the sitemap is listed. A site redesign is a good moment to review yours, and our website redesign SEO checklist shows where it fits.
Frequently asked questions
Where do I put the robots.txt file?
At the root of your domain, so it opens at the address of your site followed by /robots.txt. Google's documentation says each host has one file, and it applies only to that protocol, host and port.
Does Disallow stop a page showing in Google?
No. Google says a blocked page can still appear in results as a URL without a description. To keep a page out, use a robots meta tag that says not to index the page, or password protection.
Which rule wins when two rules match?
The most specific one, meaning the longest matching path. If an Allow and a Disallow are equally specific, Google uses the less restrictive one, which is Allow.
Does Google support crawl-delay?
No. Google's documentation says it ignores that field, so this tool does not add it.
How long does a change take to work?
Google generally caches the file for up to 24 hours, so allow a day. It may keep an older copy longer if it cannot refresh the file.
Will blocking AI crawlers work?
Only for crawlers that follow robots.txt. Crawler names change, so check each company's page, and decide whether you want to be left out of their answers.
Can I load my current file?
Yes. Paste it into the box and press Load into the form. The tool fills in the rules, bot groups and sitemap lines it finds.
Is this tool free?
Yes. It runs in your browser, needs no sign-up and does not send what you type anywhere.
More free tools
- PX to REM converter: Convert pixels to rem, with a table for any base size.
- Meta title and description checker: Count characters and preview the search result.
- OG image size checker: Check a share image against Meta's size rules.
- Color contrast checker: Test text and background colours against WCAG 2.2.