Let ChatGPT read your website

Why it matters

Every website can have a small text file called robots.txt. It tells automated visitors, called crawlers, which pages they may read. If it turns away the crawler behind ChatGPT, ChatGPT cannot read your pages. It then relies on what other websites say about you, or says nothing at all.

Websites often block AI crawlers without anyone deciding to: a rule copied from a template, a setting left on after a redesign, or a security service that blocks bots by default. This guide finds out which one applies to you.

Illustration with a fictional business.

T2Got your Starter audit? This guide fixes T2, "AI crawler access", on page 07 (Technical and agent readiness). In a free Snapshot, it is linked from your top fixes on page 06.

Reference

Which crawlers to let in

Each AI assistant uses its own crawlers. Some fetch pages to answer people's questions. Others collect text to train future AI models. To be named in answers, the first kind must be allowed. Allowing the training crawlers is your choice.

Before you start

Have these ready

• Your exact website address, including whether it starts with www.
• A login to your website editor (WordPress, Webflow, Wix) or, for a custom-built website, your developer's contact.
• If your website runs through Cloudflare, a login to your Cloudflare account. Not sure? Your developer or IT provider will know.

Steps

Check and fix your robots.txt

1. Open your robots.txt file

   In any browser, type your address followed by /robots.txt, for example yourcompany.lu/robots.txt. You will see one of three things:

   • A few lines of text. Good, go to step 2.
   • A "page not found" error. You have no file, which means every crawler is allowed. You can stop here, or create one in step 4 to add your sitemap.
   • Any other error, or a login page. Send the address to your developer: crawlers may not be able to read the file at all.

   letzscale.com/robots.txt · our own file, October 2026Copy

   User-Agent: GPTBot
   User-Agent: OAI-SearchBot
   User-Agent: ChatGPT-User
   User-Agent: ClaudeBot
   User-Agent: Claude-User
   User-Agent: Claude-SearchBot
   User-Agent: PerplexityBot
   User-Agent: Perplexity-User
   User-Agent: Googlebot
   User-Agent: Google-Extended
   User-Agent: Bingbot
   User-Agent: Applebot
   User-Agent: Applebot-Extended
   Allow: /
   
   User-Agent: *
   Allow: /
   
   Host: https://letzscale.com
   Sitemap: https://letzscale.com/sitemap.xml

   Our file names each AI crawler and allows it to read every page, then allows every other crawler too. The last line points crawlers to our sitemap, the list of our pages. The Host line is an old instruction that Google ignores, so you do not need it.

2. Look for lines that block AI

   The file is made of groups. Each group starts with User-agent: and the crawler it applies to. A * means "every crawler not named elsewhere". Under it, Disallow: / means "nothing on this site", and Allow: / means "everything".

   A crawler obeys only the group that names it most specifically. So if a group names OAI-SearchBot and says Disallow: /, a general Allow: / for everyone else does not help. Look for this pattern:

   Example of a file that blocks AI (lines in red)

   # Everyone may read the site...
   User-agent: *
   Allow: /
   # ...except these, which are locked out completely
   User-agent: GPTBot
   User-agent: OAI-SearchBot
   User-agent: ClaudeBot
   User-agent: PerplexityBot
   Disallow: /

   In this example, ChatGPT search and Perplexity cannot read the website, so they are unlikely to cite it. Also check for Disallow: / directly under User-agent: *. That blocks everyone, including Google and Bing, and is a common leftover from a website that was built on a test server.

3. Check the settings that change the file for you

   On most platforms you do not edit the file directly; a setting writes it. Check the one that applies to you:

   WordPress

   Settings → Reading → Search engine visibility must be unticked. If you use an SEO plugin (Yoast, Rank Math), its tools section has a robots.txt editor.

   Webflow

   Site settings → SEO → Indexing. Edit the rules, save, then publish the site. Webflow help

   Wix

   Dashboard → SEO & GEO → Tools and settings → Robots.txt Editor → View File. Wix help

   Using Cloudflare?

   Cloudflare is a service many websites use for speed and security. It can block AI crawlers at the door, even when robots.txt allows them. In your Cloudflare dashboard, open Security Settings, then Configure AI bot policies. Keep the Search category on "Allow". Since 15 September 2026, new Cloudflare domains block training and agent bots on pages with ads by default. Cloudflare docs

4. Write or update the file

   If you edit the file yourself, this is the version we recommend. Replace the sitemap address with yours. Most website builders create the sitemap for you at /sitemap.xml. Lines starting with # are comments. Remove the # from the training lines only if you want to opt out of AI training.

   robots.txtCopy

   # Search engines and AI assistants may read every page
   User-agent: *
   Allow: /
   
   # Optional: opt out of AI model training only
   # User-agent: GPTBot
   # User-agent: ClaudeBot
   # User-agent: Applebot-Extended
   # Disallow: /
   
   Sitemap: https://www.yourcompany.lu/sitemap.xml

   • Save it as plain text named exactly robots.txt, never from Word, which can add characters that break it.
   • It must sit at the top of your website: yourcompany.lu/robots.txt, not in a folder.
   • Each address needs its own file: shop.yourcompany.lu is separate from yourcompany.lu.

Check it worked

Confirm what crawlers now see

Open yourcompany.lu/robots.txt again in a private browser window, so you see the live version and not a saved copy. Then confirm with the search engines themselves:

• Google Search Console → Settings → robots.txt shows the version Google last fetched and any problems it found. No Search Console yet? Guide 03 sets it up.
• Bing Webmaster Tools has a robots.txt tester that shows whether a page is allowed for Bingbot, which Copilot depends on.

search.google.com/search-console · Settings · robots.txt

Give it a day. OpenAI says its search systems can take about 24 hours to pick up a change; other crawlers follow their own schedule.

Common mistakes

What usually goes wrong

• A test-site block that went live. Disallow: / under User-agent: * keeps everyone out, Google included.
• Fixing the wrong address. The website runs on www.yourcompany.lu but the file was changed on yourcompany.lu, or the other way round.
• Blocking Google-Extended to protect your Google rankings. It has no effect on Google Search. It only controls how Google uses your content for Gemini.
• Treating robots.txt as a lock. It is a public request: well-behaved crawlers follow it, others do not. Never use it to hide private pages.
• Forgetting the firewall. A security service can block a crawler that robots.txt allows. If the file looks right and ChatGPT still cannot read you, check Cloudflare or ask your host.