See AI crawler visits in your server logs

Why it matters

Guide 02 opens your robots.txt to AI crawlers and Guide 11 makes sure your firewall lets them through. Both are settings. Your logs are the proof: they show whether the crawlers actually came, which pages they asked for, and what your website answered.

Every time anything asks your website for a page, a person or a bot, the server writes one line in its access log. Each line includes the name the visitor gives itself, such as OAI-SearchBot for ChatGPT search, and a three-digit answer code. Filter the log for AI crawler names and you can see, page by page, whether they were let in.

One limit to keep in mind: a log shows that a crawler read a page. It does not show whether an assistant later used that page in an answer.

Reference

How to read one log line

Most web servers write lines in the same "combined" layout: it is the default in nginx and a standard format in Apache. Here is one line, split into its parts:

192.0.2.14 - - [23/Oct/2026:06:12:41 +0200] "GET /services HTTP/1.1" 200 18342 "-" "... compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot"

What the status codes mean

Reference

The names to look for

These are the names AI assistants and search engines put in their requests, and how each company lets you check a visit is really theirs.

Two names you will never find in a log: Google-Extended and Applebot-Extended. They are switches in robots.txt, not crawlers. Google crawls with its normal user agents, and Apple says Applebot-Extended does not crawl web pages.

Before you start

Have these ready

• Your domain name, and whether it runs through Cloudflare (Guide 11, step 1, shows how to check).
• A login to Cloudflare or to your hosting control panel, or a developer with access to the server.
• For steps 3 and 5: a terminal on the server (Linux or macOS commands below).

Steps

Find, filter and check the visits

1. Choose where to look

   Your logs live in one of three places, depending on how your website is hosted. Use the first one that applies.

   On Cloudflare

   Cloudflare's AI Crawl Control already counts AI crawler visits for you, on all plans. Go to step 2.

   Hosting panel

   Many hosts let you download the raw log. In cPanel it is Metrics → Raw Access. Other panels call it Logs or Statistics. Go to step 3.

   Your own server

   Your developer reads the nginx or Apache log directly on the server. Go to step 3.

2. On Cloudflare: read AI Crawl Control

   In the Cloudflare dashboard, open your domain, then AI Crawl Control.

   • Overview tab: total requests, allowed and unsuccessful requests over time, the most common status code and the most popular path.
   • Crawlers tab: one row per crawler with its Requests and its Action (Allow or Block). The menu on each row has View Metrics.
   • Metrics tab: the Status code distribution chart splits answers into 2xx, 3xx and 4xx (including 403 blocked) and 5xx. Use the Crawler filter to see one bot at a time, and Most popular paths to see which pages it read.

   Look at OAI-SearchBot, Claude-SearchBot and PerplexityBot first. You want mostly 2xx. You can export any chart with Download CSV.

   On the free plan, Cloudflare recognises crawlers by the name they announce, so the view shows well-known bots that identify themselves. Referral figures (visitors arriving from AI answers) need a paid plan.

3. Developers: filter the raw log for AI crawlers

   First find the file. In nginx, the access_log line in the configuration names it; this prints the whole configuration so you can search it:

   Terminal · find the nginx access logCopy

   sudo nginx -T | grep access_log

   In Apache, look for the CustomLog line in the configuration. A relative path is relative to the ServerRoot. From cPanel's Raw Access, you download a compressed .gz file per domain.

   Then keep only the lines from AI crawlers and search engines. Replace access.log with your path. The -i matters: Bing writes its name in lower case (bingbot).

   Terminal · AI crawler lines onlyCopy

   grep -iE 'OAI-SearchBot|ChatGPT-User|GPTBot|Claude-SearchBot|Claude-User|ClaudeBot|PerplexityBot|Perplexity-User|bingbot|Googlebot|Applebot' access.log

   Older days are often kept in separate files, some compressed. zgrep takes the same options and reads .gz files too, so zgrep -iE '...' access.log* searches all of them.

   The result looks like this:

   Filtered log for kastell-conseil.example (user agents shortened)

   192.0.2.14 - - [23/Oct/2026:06:12:41 +0200] "GET /services HTTP/1.1" 200 18342 "-" "... compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot"
   192.0.2.31 - - [23/Oct/2026:07:40:03 +0200] "GET /fr/services HTTP/1.1" 429 162 "-" "... Claude-SearchBot ..."
   192.0.2.58 - - [23/Oct/2026:09:02:17 +0200] "GET /contact HTTP/1.1" 200 9120 "-" "... compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)"
   192.0.2.77 - - [23/Oct/2026:10:15:52 +0200] "GET / HTTP/1.1" 403 153 "-" "... compatible; GPTBot/1.4; +https://openai.com/gptbot"
   192.0.2.90 - - [23/Oct/2026:11:28:09 +0200] "GET /equipe HTTP/1.1" 200 14877 "-" "... bingbot ..."
   198.51.100.23 - - [23/Oct/2026:12:01:44 +0200] "GET /services HTTP/1.1" 200 18342 "-" "... compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot"

   Illustration with a fictional business. The IP addresses are reserved example addresses.

   For a quick count of answers per crawler, add the status column. In the combined format it is the ninth field:

   Terminal · status codes for one crawlerCopy

   grep -i 'OAI-SearchBot' access.log | awk '{print $9}' | sort | uniq -c

4. Read what your server answered

   Go through the filtered lines with the status code table above. In the Kastell Conseil example:

   • OAI-SearchBot, PerplexityBot and bingbot got 200. ChatGPT search, Perplexity and Bing could read those pages.
   • Claude-SearchBot got 429. A rate limit cut it off on the French services page. That needs fixing: see Guide 11, step 4.
   • GPTBot got 403. That is fine if you chose to block training crawlers (Guide 13). It does not affect ChatGPT search, which uses OAI-SearchBot.

   No lines at all from a crawler can mean it has not come yet, or that something turned it away before it reached your server. Do not expect ChatGPT-User, Claude-User or Perplexity-User on a schedule: they only visit when someone asks the assistant about a page.

5. Developers: check the bot is real

   The last line in the example calls itself OAI-SearchBot, but anyone can. Google warns that its own user agent is often copied by other crawlers. Each company gives you a way to check.

   OpenAI, Anthropic, Perplexity (and Bing, Apple, Google): published IP lists. Check whether the address from the log is inside the provider's list. Perplexity recommends checking both the name and the IP address. This command downloads a list and answers True or False:

   Terminal · is this IP on OpenAI's search crawler list?Copy

   curl -sL https://openai.com/searchbot.json | python3 -c 'import ipaddress,json,sys; ip=ipaddress.ip_address(sys.argv[1]); print(any(ip in ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix")) for p in json.load(sys.stdin)["prefixes"]))' 198.51.100.23

   Swap the list for the crawler you are checking: https://openai.com/chatgpt-user.json, https://openai.com/gptbot.json, https://claude.com/crawling/bots.json, https://www.perplexity.com/perplexitybot.json, https://www.perplexity.com/perplexity-user.json, https://www.bing.com/toolbox/bingbot.json, http://search.developer.apple.com/applebot.json, or Google's https://developers.google.com/static/crawling/ipranges/common-crawlers.json. The lists change, so always download the current one. In the Kastell Conseil example, the last visitor's address is not on OpenAI's list, so the answer is False.

   Google, Bing and Apple: reverse DNS. Look up the name behind the IP address, then look up that name and confirm it points back to the same address:

   Terminal · reverse, then forward lookupCopy

   host 66.249.66.1
   # then run host again on the name it returns

   • Googlebot: the name must end in googlebot.com, google.com or googleusercontent.com.
   • bingbot: the name must end in search.msn.com. Bing also has a Verify Bingbot page where you paste an IP address.
   • Applebot: the name must end in applebot.apple.com.

   In both methods, the second lookup must give back the address you started with. If it does not, or the IP is not on the list, the visitor is not the crawler it claims to be. Leave it out of your counts; you can block it, as Guide 11 explains.

   Behind Cloudflare? Your server sees Cloudflare's addresses

   When Cloudflare sits in front of your website, your server logs Cloudflare's IP address, not the crawler's. Cloudflare passes the original address in a header called CF-Connecting-IP, and your developer can set the server to log it. Until then, use AI Crawl Control (step 2) instead of IP checks.

Check it worked

Confirm the crawlers get in

After any change from Guide 02 or Guide 11, look again a few days later. Crawlers come back on their own schedule, and OpenAI says its search systems can take about 24 hours to pick up a robots.txt change.

Common mistakes

What usually goes wrong

• Searching without -i. Bing's crawler is written bingbot, in lower case. A case-sensitive search for "Bingbot" finds nothing.
• Reading only today's file. Many servers start a new log file every day or week and compress the older ones. Search them all, or you may think a crawler never came.
• Trusting the name. A line that says OAI-SearchBot is not proof. Check the IP address before you count it.
• Checking IPs behind Cloudflare. Without CF-Connecting-IP logging, every visit shows a Cloudflare address and no crawler will pass the IP check.
• Looking for Google-Extended or Applebot-Extended. They never appear in logs. Google and Apple use what their normal crawlers, Googlebot and Applebot, fetch.
• Reading a visit as a recommendation. A 200 means the crawler read the page. It does not mean the assistant will use or recommend it.