Can ChatGPT read your website?

Training crawlers: should you block GPTBot, ClaudeBot and Google-Extended?

AI companies send search crawlers, which read your pages for today's answers, and training crawlers, which collect text for future models. In 15 minutes you decide whether to opt out of training, write the decision down and add the right lines to your robots.txt, without dropping out of ChatGPT's search answers.

Published 24 October 2026 · By the Letzscale team · 6 min read

On this page

Why it matters
The four training controls
A simple rule
Make the choice and apply it
Confirm the file says what you decided
What usually goes wrong

Time: 15 minutes
Difficulty: Easy
Who: The business owner decides; whoever edits your website applies it
You need: A browser and access to your robots.txt file
Checked: 24 October 2026

Part of the Letzscale guides: free, step-by-step fixes that help ChatGPT and other AI assistants find, understand and recommend your business. Not sure where you stand yet? Start with Guide 01: Check what ChatGPT says about your business.

Why it matters

AI companies send two kinds of automated visitors, called crawlers, to your website. Search crawlers read your pages so an assistant can use them in live answers today. Training crawlers collect text that may be used to build future AI models.

The two are controlled separately. You can say no to training and still let ChatGPT, Claude and Siri read your pages for their answers. Many websites block training crawlers by copying a rule from somewhere, without anyone deciding it. This guide helps you make the choice on purpose, write it down, and put the right lines in your robots.txt file.

New to robots.txt? Guide 02 explains what the file is and how to find it on your website.

Reference

The four training controls

These are the names to use in robots.txt if you want to opt out of AI training. Everything in the table comes from each company's own page, checked on 24 October 2026.

Company	Name in robots.txt	What blocking it does	What it does not change
OpenAI ChatGPT	`GPTBot`training	OpenAI says blocking GPTBot tells it your content should not be used to train its AI models.	ChatGPT search. That is `OAI-SearchBot`, and OpenAI says each setting is independent of the others.
Anthropic Claude	`ClaudeBot`training	Anthropic says blocking ClaudeBot signals that your site's future material should be left out of its AI training data.	Claude's search and page visits. Those are `Claude-SearchBot` and `Claude-User`, listed separately.
Google Gemini	`Google-Extended`	A switch, not a separate crawler. It controls whether content Google crawls may be used to train future Gemini models and to ground answers in the Gemini apps and Google's Vertex AI, which means giving the model content from the Google Search index when someone asks a question.	Google Search. Google says it does not affect whether you are included in Google Search and is not a ranking signal.
Apple Siri, Apple Intelligence	`Applebot-Extended`training	A switch, not a crawler. It decides whether what Applebot read may be used to train Apple's foundation models, which power generative AI features such as Apple Intelligence.	Apple's search. Apple says pages that block Applebot-Extended can still appear in search results, and the rule is not used for ranking.

Google-Extended is different

For OpenAI, Anthropic and Apple, blocking the training name leaves their search features alone. Google-Extended covers two things at once: Gemini training and "grounding" in the Gemini apps. Blocking it does not touch Google Search, but Google may use your pages less when answering in Gemini. Decide on it separately.

What about content already collected?

Anthropic's page describes blocking ClaudeBot as covering your site's future material. None of the four pages says that blocking removes content collected before you changed the file. Treat the change as a choice about what happens from now on.

Decide

A simple rule

Most Luxembourg service businesses (cafés, trades, fiduciaries, law firms, consultancies) want AI assistants to know their services, prices, opening hours and contact details. For them, allowing everything is the simplest choice. Blocking training makes sense when your text itself is what you sell, or when a client or company policy says so.

Your situation	Our suggestion
Your website describes your business: services, team, prices, contact.	Option A: allow everything.
You publish content you sell or license (paid research, course material, articles), or a client contract or company policy forbids AI training.	Option B: block GPTBot, ClaudeBot and Applebot-Extended. Keep Google-Extended allowed.
As in B, and you accept that Gemini may use your pages less in its answers.	Option C: block all four.

Examples: Café Ribeiro and Fiduciaire Veldrin would pick A. Kastell Conseil, which sells paid market reports on its website, might pick B.

Steps

Make the choice and apply it

Write the decision down

One line is enough. Keep it where your team keeps website notes, so nobody changes the file later without knowing why it says what it says.
Decision record
```
AI training crawlers: we allow / we block (GPTBot, ClaudeBot, Applebot-Extended, Google-Extended), decided on [date] by [role].
```
Open your robots.txt

Type your address followed by /robots.txt, for example yourcompany.lu/robots.txt. Look for any group that names GPTBot, ClaudeBot, Google-Extended or Applebot-Extended. If you find one you did not decide on, this is your chance to fix it. Guide 02 shows where WordPress, Webflow and Wix let you edit the file.

Add the lines for your choice

Lines starting with # are comments, which crawlers ignore. Replace the sitemap address with yours. Several User-agent lines can share one rule, as below.

Option A · allow everything

# Search engines and AI assistants may read every page
User-agent: *
Allow: /

# Optional: opt out of AI model training only
# User-agent: GPTBot
# User-agent: ClaudeBot
# User-agent: Applebot-Extended
# Disallow: /

Sitemap: https://www.yourcompany.lu/sitemap.xml

Option B · block training at OpenAI, Anthropic and Apple

# Search engines and AI assistants may read every page
User-agent: *
Allow: /

# Opt out of AI model training (search is not affected)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Applebot-Extended
Disallow: /

Sitemap: https://www.yourcompany.lu/sitemap.xml

Option C · also block Google-Extended

# Search engines and AI assistants may read every page
User-agent: *
Allow: /

# Opt out of AI model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Applebot-Extended
# Also limits use in Gemini answers; Google Search is not affected
User-agent: Google-Extended
Disallow: /

Sitemap: https://www.yourcompany.lu/sitemap.xml

Do not add OAI-SearchBot, Claude-SearchBot, Claude-User, Applebot or Googlebot to the blocked group. Those are the search side. OpenAI says sites that block OAI-SearchBot are not shown in ChatGPT search answers.

Repeat for every subdomain

The file only covers the address it sits on. shop.yourcompany.lu needs its own robots.txt at its top level. Anthropic asks for the same: add the rule for every subdomain you want to opt out.

Check it worked

Confirm the file says what you decided

Open yourcompany.lu/robots.txt in a private browser window, so you see the live version and not a saved copy. Read the groups once more against your decision record.

Done when

Your decision is written down with a date and a role.
The live robots.txt shows plain text and matches Option A, B or C.
No group blocks *, Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot or Applebot.
Every subdomain you care about has its own file.

Common mistakes

What usually goes wrong

Blocking the search crawler instead of the training one. OAI-SearchBot and Claude-SearchBot decide whether you can appear in answers. Blocking them does not stop training; it removes you from search.
Blocking Google-Extended to protect your Google rankings. Google says it has no effect on Google Search inclusion or ranking. It does affect Gemini.
Assuming Google-Extended is only about training. It also covers grounding in the Gemini apps, so blocking it can mean fewer Gemini answers that use your pages.
Blocking by IP address instead of robots.txt. Anthropic warns that blocking its IP addresses may not work reliably, because it stops its crawler from reading your robots.txt.
Copying a rule nobody decided on. A template or plugin adds a block, and a year later nobody knows why. The decision record prevents this.

Next guide14 · Content AI cannot read: cookie walls, logins, PDFs and text in images →

1 hour · Easy

Sources, checked 24 October 2026