robots.txt Explained: What It Does, What to Put In It, and How to Handle AI Crawlers

Key takeaways

• robots.txt asks crawlers which addresses they may fetch. It is a plain text file at the root of your domain, such as example.com/robots.txt.
• It does not hide pages from Google. Google says a blocked URL can still be indexed if other pages link to it. To keep a page out of search, use no-index or a password.
• Most small business sites need very little in it. Often just "allow everything" and a sitemap line.
• AI companies use separate bot names for training and for search. OpenAI's GPTBot is for training and OAI-SearchBot for ChatGPT search; Anthropic splits ClaudeBot, Claude-SearchBot and Claude-User.
• Google-Extended controls Gemini training and grounding, not Google Search. Google says it does not affect inclusion or ranking in Search.
• Many platforms write the file for you. Webflow, Shopify and Wix let you edit it; Squarespace offers tick boxes instead.
• One stray line can block your whole site. "Disallow: /" left over from a staging site is the classic mistake.

The short answer

robots.txt is a short text file that tells well-behaved bots which parts of your site they may crawl. For a typical business site, it should allow everything and point to your sitemap. Add rules only for a real reason: to keep crawlers out of internal search results or endless filter pages, or to opt out of specific AI training bots. Never use it to hide a private page, because it is a public request, not a lock, and Google can still list a blocked address. If you want to build one, our free robots.txt generator writes the file from your choices.

Who is writing: Amariq designs and builds websites and sets up their SEO. Every rule and bot name below comes from the official page named next to it, read on 10 November 2026. We checked our own file too: amariq.com's robots.txt allows all crawlers and lists the sitemap.

What ranks for this search, and what it leaves out

On 10 November 2026 the first page of Google (US, English) for "robots.txt" was mostly reference material: Google Search Central's introduction and its "create and submit" guide (updated 21 November 2025), Wikipedia, Cloudflare's learning page, robotstxt.org, Google's own robots.txt file, a Search Console help page, and short videos, several about AI bots.

They explain what the file is and how a rule is written. What a site owner still has to piece together elsewhere: a ready file for an ordinary business site, a current list of AI bot names with what blocking each one actually does, and where the setting lives on Webflow, Shopify, Wix or Squarespace. This guide puts those in one place.

What robots.txt does, and what it does not

Google's introduction says the file is used mainly to avoid overloading a site with requests, and that it is not a way to keep a page out of Google.

That second row matters most. Google says that if a blocked page is linked from elsewhere, its address and the link text can still show in results, and that a blocked page can appear without a description. To keep a page out, Google says to use a no-index rule, password protection or remove the page. And a no-index rule only works if crawlers are allowed to fetch the page and see it.

How the file works

Google's guide to creating the file sets the basics:

• Name and place: exactly robots.txt, one per site, at the root of the host. Rules on www.example.com do not cover shop.example.com or the http version.
• Format: a plain UTF-8 text file. Google warns against word processors, which can add curly quotes.
• Groups: each group starts with a User-agent line naming the bot, followed by Allow and Disallow lines with paths that start with a slash.
• Default: anything not disallowed may be crawled.
• Case matters: /File.html and /file.html are different paths.
• Comments: anything after # is ignored.
• Sitemap: a full address, such as Sitemap: https://www.example.com/sitemap.xml.

Google adds one detail people miss: User-agent: * matches all crawlers except Google's AdsBot crawlers, which must be named on their own.

Sample files you can start from

These are illustrations based on the rules above. Each line in the left column is one line of the file, in order. Swap in your own domain and paths, and test before you publish.

1. A typical business site (most sites need only this)

Google notes that the Allow line could be left out with the same result, because everything is allowed by default. Many people keep it for clarity.

2. A site with internal search and a members area

The members area still needs a login to be private. The rule only saves crawl effort.

3. Search allowed, AI training opted out

Put each User-agent and its Disallow or Allow on separate lines, with a blank line between groups. What each name controls is in the next section.

4. A staging or test site that should not be crawled

Use this only on the test copy, ideally together with a password. Copying this file to the live site is how businesses vanish from search.

AI crawlers: which name does what

AI companies now publish separate names for training, search and user-requested visits. Blocking the wrong one can remove you from AI answers without stopping training, or the other way round. Each row comes from the company's own page, read on 10 November 2026.

Three more points from those pages:

• Anthropic says it supports the non-standard Crawl-delay rule, and that blocking its IP addresses may not work as an opt-out because it stops the bot from reading your robots.txt. It also says that, from 30 October 2026, it has begun honouring a general "no AI training" preference signal based on IETF work, with rules that name ClaudeBot taking priority.
• Google says it intends to support the same shared training preference signal, written as a user-agent line, in early 2027.
• OpenAI says the settings are independent: you can allow OAI-SearchBot and block GPTBot.

Whether to block training bots is a business decision. Squarespace's help page frames it well: blocking may reduce traffic from AI answers, and it does not remove anything already collected. If you want to see how often AI answers already use your pages, Bing now reports it; our Bing Webmaster Tools guide covers the AI Performance report.

Where to edit robots.txt on your platform

Google's own guide says the same about hosted builders: on services such as Wix you may not be able to edit the file directly, and should look for a search settings page instead.

Mistakes we see most often

1. Disallow: / on the live site. Usually copied from a staging build or left on by a "hide from search engines" setting.
2. Blocking CSS or JavaScript folders. Google says not to block resources if the page is harder to understand without them.
3. Using robots.txt to hide a page. The address can still be indexed; use no-index instead.
4. Blocking a page that carries no-index. If the crawler cannot fetch it, it never sees the no-index.
5. Wrong file location or name. It must be /robots.txt at the root of each host and subdomain.
6. Rules with wrong case or no leading slash.
7. Blocking OAI-SearchBot or Claude-SearchBot by accident while meaning to stop training only.

How to check your file

1. Open yoursite.com/robots.txt in a private browser window. You should see plain text.
2. Read every Disallow line and ask what it protects and why.
3. In Google Search Console, open the robots.txt report. Google says it shows whether Google can process your file.
4. For important pages, use URL Inspection to confirm crawling is allowed.
5. After any change, check again in a few days.

If your pages are not showing in Google at all, robots.txt is one of the first things our not-on-Google checklist looks at.

How Amariq can help

On every site we build, we set robots.txt, the sitemap and no-index rules on purpose, check that the staging rules did not travel to the live site, and agree with you which AI bots, if any, to opt out of. If you are not sure what your current file is doing, our website audit checklist covers it, or send us the address. We work on Webflow, Framer, WordPress, Shopify, Wix, Squarespace or a custom stack. Each project is priced from its scope, never by the hour, with 40% to start, 30% at design approval and 30% before launch. Once the final payment is made, the design, copy and code belong to you, and your domain, hosting and analytics accounts sit in your name.

Frequently asked questions

What is a robots.txt file?

A plain text file at the root of a website that tells crawlers which addresses they may fetch. Google says it is used mainly to manage crawler traffic.

Do I need a robots.txt file?

Not strictly. Google says everything is allowed by default. Most sites still have one, if only to list the sitemap.

Does robots.txt stop Google from indexing a page?

No. Google says a blocked address can still be indexed if it is linked from elsewhere. Use no-index or a password to keep a page out.

How do I block ChatGPT in robots.txt?

OpenAI publishes separate names. GPTBot is for training; OAI-SearchBot is for ChatGPT search. Disallow the one you mean. OpenAI says ChatGPT-User visits are started by users and robots.txt may not apply.

Does blocking Google-Extended hurt my Google rankings?

Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal.

Where is robots.txt on Shopify?

Shopify generates it. To change it, add a robots.txt.liquid template in the theme's Templates folder; Shopify says this replaces the generated file.

Can bad bots ignore robots.txt?

Yes. Google says the file cannot enforce behaviour and it is up to each crawler to obey. Private content needs a password.