Check if Cloudflare or your firewall blocks AI bots

Why it matters

Your robots.txt file asks crawlers politely which pages they may read. Guide 02 shows how to set it. But a request is not a lock. A firewall is: if a security service in front of your website turns a crawler away, it never reaches your pages, whatever robots.txt says.

This happens without anyone deciding it. Cloudflare, the most common of these services, has settings that block AI bots, and since 15 September 2026 new Cloudflare domains block some AI bots on pages with ads by default. Hosting companies and WordPress security plugins can block "bad bots" too. This guide checks whether any of that applies to you.

Illustration with a fictional business.

Reference

Which bots to let through

Each AI company runs separate bots for search and for training. For your business to be read when someone asks, the search bots must get through. Blocking training bots is your choice; Guide 13 helps you decide.

Google's AI answers and Microsoft Copilot use the normal Googlebot and Bingbot, which firewalls rarely block. If they are blocked, your website is missing from Google and Bing altogether, and Guide 03 will show it.

Before you start

Have these ready

• Your domain name, for example yourcompany.lu.
• If you use Cloudflare, a login with access to the domain.
• Otherwise, a login to your hosting control panel, or your web developer's contact.

Steps

Find and fix the block

1. Find out if you use Cloudflare

   Look up your domain's nameservers, the servers that say where your website lives. Any free "NS lookup" or "WHOIS" website will show them, or on a Mac or Linux computer type this in Terminal:

   TerminalCopy

   dig ns yourcompany.lu @1.1.1.1

   On Windows, use nslookup -type=ns yourcompany.lu 1.1.1.1.

   • Names ending in .ns.cloudflare.com (or foundationdns.com): you use Cloudflare. Go to step 2.
   • Anything else (your host's or registrar's names): Cloudflare is probably not in front of your site. Go to step 4. Some larger setups use Cloudflare without its nameservers; if in doubt, ask your developer.

2. Check the AI bot policy in Cloudflare

   This is the setting that blocks AI bots by category. In the Cloudflare dashboard, open your domain, then Security Settings → Configure AI bot policies. Each category has three options: "Block (on all pages)", "Block on pages with ads" and "Allow (do not block)".

   • Search: set to Allow (do not block). These are the crawlers that read your content to answer questions about it later.
   • Training: your choice. Blocking it does not stop search crawlers.
   • Agent: bots acting in real time for a person, for example fetching your page when someone asks about it. Most businesses should allow them.

   Cloudflare notes that crawlers which do both search and training are blocked whenever you block training. If you see an older option called Block AI bots, Cloudflare is phasing it out in favour of these policies.

3. Check each crawler in AI Crawl Control

   Cloudflare's AI Crawl Control page, available on all plans, lists the AI crawlers visiting your domain and lets you allow or block each one. Open your domain, go to AI Crawl Control, then the Crawlers tab. In the Action column:

   • Set OAI-SearchBot, Claude-SearchBot and PerplexityBot to Allow.
   • Leave GPTBot and ClaudeBot blocked if you decided not to take part in AI training.

   Blocking a crawler here creates a firewall (WAF) rule on your domain, so also look through your WAF custom rules for a stricter rule someone added by hand. On the free plan, Cloudflare recognises crawlers by the name they announce, so only well-known bots appear in the list.

   Also check Bot Fight Mode

   Cloudflare's Bot Fight Mode challenges traffic that looks like a bot, and Cloudflare says it cannot be bypassed with your own firewall rules. You find it under Security Settings, filtered by Bot traffic. If AI crawlers are blocked and you cannot find a rule that explains it, this is the next setting to look at.

4. Not on Cloudflare: check your host and plugins

   Other services block bots too: some hosting companies, other CDNs, and WordPress security plugins with options like "block bad bots" or rate limiting. The quickest way to find out is to look at what your server answered.

   Hosting panel

   Open the access log (often under Statistics, Logs or Metrics). Search for OAI-SearchBot, Claude-SearchBot and PerplexityBot.

   WordPress plugin

   Open the security plugin's settings and look for bot blocking, rate limiting or a "blocked" log. Search it for the same names.

   Developer

   Send your developer this guide and ask: "Do we block or rate-limit OAI-SearchBot, Claude-SearchBot or PerplexityBot anywhere?"

   In a log line, the three-digit number after the request is the answer your server gave. 200 means the page was sent. 403 means "forbidden": the bot was refused. 429 means "too many requests": it was slowed down or cut off. Guide 16 shows how to read logs in more detail.

   Example log line where a search crawler is refused (shortened)

   "GET /services HTTP/1.1" 403 ... "compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot"

5. Developers: allow by IP list, not by name alone

   Anyone can write "OAI-SearchBot" in a request, so an allow rule based only on the name lets impostors in too. OpenAI and Perplexity publish the IP addresses their bots use, and Perplexity recommends checking both the name and the IP address. If you write your own allow rule:

   • Match the bot's name and an address from its official list: https://openai.com/searchbot.json, https://openai.com/chatgpt-user.json, https://www.perplexity.com/perplexitybot.json, https://www.perplexity.com/perplexity-user.json.
   • Fetch the lists automatically and often: the providers change them.

   Anthropic publishes its crawlers' IP addresses too, at claude.com/crawling/bots.json, but advises against blocking by IP, because it can stop its bots from reading your robots.txt. For Claude, rely on robots.txt and the Cloudflare setting.

Check it worked

Confirm the bots get through

• On Cloudflare, open AI Crawl Control again after a day or two. The search crawlers should show as allowed, with requests coming through.
• Elsewhere, search your access log again: new lines from OAI-SearchBot, Claude-SearchBot and PerplexityBot should end in 200, not 403 or 429.

Give it time. OpenAI says its search systems can take about 24 hours to pick up a robots.txt change, and crawlers only return on their own schedule.

Common mistakes

What usually goes wrong

• Fixing robots.txt and stopping there. If the firewall blocks a bot, the file is never even read.
• Blocking every AI bot to stop training. One switch for "AI" also turns away the search crawlers that let assistants read you. Block training only.
• Allowing by name alone. A rule that trusts any request calling itself OAI-SearchBot also lets in fake bots. Check the official IP lists.
• Forgetting the defaults. A domain added to Cloudflare after 15 September 2026 starts with some AI bots blocked on pages with ads. Check the settings even if no one changed them.
• Looking only at Cloudflare. A WordPress plugin or the host's own protection can block bots behind Cloudflare too.