jsonLd && ( ) Skip to content
All posts

How to Check if AI Crawlers Can Reach Your Site

Three layers decide whether ChatGPT, Claude or Perplexity can read your pages. How to check each one, and how to tell real crawlers from fake ones.

Updated

If ChatGPT, Claude or Perplexity can’t read your pages, they can’t quote them or link to them in their answers. We wanted to know whether that was true for nettingale.com before we started writing here, and it turned into a small checklist that is worth running on any site. Three layers can block an AI crawler, and they have to be checked separately: the robots.txt file, your own server, and the CDN or firewall in front of it.

Know which crawler does what

Most AI companies now run more than one crawler, and they are blocked or allowed independently. OpenAI uses GPTBot for collecting training data, OAI-SearchBot for its search index, and ChatGPT-User when a person asks ChatGPT to open a page. Anthropic has the same split with ClaudeBot, Claude-SearchBot and Claude-User, and Perplexity crawls as PerplexityBot. If you want to show up in AI answers but not in training data, the search and user crawlers are the ones to allow.

Google works differently. Google-Extended is not a crawler at all, it’s a token you use in robots.txt to say whether pages that Googlebot already fetches may be used for Google’s AI models. When we first ran our own checks we sent requests with Google-Extended as the user agent, and the result meant nothing, because no real crawler ever sends it.

Check robots.txt and your server

Start with your robots.txt, at https://yoursite.com/robots.txt. Ours allows everything:

User-agent: *
Allow: /

A rule like User-agent: GPTBot followed by Disallow: / blocks that one crawler and nothing else, so read through the file for each name you care about. SEO and security plugins sometimes add these lines without asking.

Then check that your server answers normally when a request says it is one of these crawlers:

for ua in GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-SearchBot PerplexityBot; do
  printf "%-18s" "$ua"
  curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; $ua)" https://yoursite.com/
done

Every line should show 200. A 403 usually means a security plugin or a server rule is blocking that name. Our own rule against AI crawlers flooding shop carts only refuses cart URLs, so a normal page should still answer 200. It is part of the protection that comes with every site we host, and it leaves normal pages open to crawlers.

Check your CDN or firewall

This is the layer the curl test can’t fully cover. Cloudflare and similar services recognize the real AI crawlers by the IP addresses they come from, not only by their user agent, so a request from your laptop that claims to be GPTBot isn’t treated the same way as the real one. Cloudflare also has settings that block AI crawlers by category, and on newer zones some of them are switched on by default. Look under the Security and AI crawler settings for your domain in the Cloudflare dashboard, and check what is blocked for search, training and user-initiated fetches.

After the settings look right, give it a few days and check whether the crawlers actually arrive. Your server’s access logs, or the AI crawler report in Cloudflare, will show requests from these user agents if they can reach you.

Real crawlers and fake ones

Once the checks were in place we started counting which crawlers actually visit nettingale.com, and most requests that carry an AI crawler’s name don’t come from that crawler. Cloudflare marks a request as a verified bot when it comes from the network the crawler’s owner uses, so we compared those requests with the ones that only used the name. Over four days in October, for the eight crawlers that Cloudflare verified at least once on our site, there were about six requests using the name for every real one, and 83% of the fake ones asked for pages that don’t exist here, which is what a vulnerability scanner does and not what a crawler following links does.

The split was very uneven. bingbot only ever showed up for real, while ChatGPT-User was imitated more than a hundred times for every real visit. That matters for the checks above in two ways, because a firewall or plugin rule that matches only the user agent blocks the scanners and the real crawler alike, and your access logs make AI crawlers look far busier than they are.

To tell them apart, check where the request comes from. OpenAI publishes the IP ranges of its three crawlers at openai.com/gptbot.json, openai.com/searchbot.json and openai.com/chatgpt-user.json, Google lets you confirm Googlebot with a reverse DNS lookup, and Cloudflare does this matching for you and shows the result as a verified bot. Perplexity is a special case, because Cloudflare removed it from its verified bot list in August 2025 after finding it crawling under other names, so an unverified Perplexity request on Cloudflare doesn’t prove that it’s fake.

Before you block a crawler or count its visits, check that the requests come from the crawler’s own network.

See all posts