AI Crawlers in Your Analytics: GPTBot, ClaudeBot, and the Traffic You Did Not Invite

The visitors that never buy anything

A store owner opens the analytics, sees traffic climbing, and then checks sales and finds nothing has changed. The new visitors are not shoppers who decided not to buy. They are crawlers, and the fastest growing kind in 2026 is the AI crawler: automated programs that AI companies operate to harvest training data or fetch live content for user-facing answers. Cloudflare's breakdown by purpose found training responsible for nearly 80 percent of AI bot crawling, with GPTBot plus Amazonbot alone accounting for over 40 percent of crawling traffic. Your server pays the bandwidth for every one of those visits, and the referral may never come, because a training crawler takes the page and sends nobody back.

The trade is different from a search crawler. Googlebot indexes your content so a human can find it and click through, and the exchange is fair: you give up content, you get a visitor. An AI training crawler often breaks that trade. It uses your content to train a model or to generate an answer inside the AI product, frequently without sending the reader to you at all. That is why the first step is not blocking everything. It is sorting the crawlers by what they do for you.

Training crawlers and search crawlers are different bots

OpenAI runs two bots that look similar and do opposite jobs. GPTBot collects content that may train OpenAI's models. OAI-SearchBot indexes content for ChatGPT's live search results. Anthropic splits the same way: ClaudeBot collects content for training, while Claude-SearchBot indexes for search and Claude-User fetches a page when a Claude user asks for it. Google-Extended is not a crawler at all. It is a control token that tells Google whether its content may be used for generative AI training. Blocking GPTBot tells OpenAI your content should not be used for training, and it has no measurable impact on Google search rankings. Blocking OAI-SearchBot removes your pages from ChatGPT search answers entirely, including the citation that could have sent a reader your way.

The distinction is the whole decision, and it is easy to get wrong in both directions. Research cited by Mersel found roughly 27 percent of B2B SaaS and ecommerce websites accidentally blocking major LLM crawlers through CDN-level rules, often without knowing it. Meanwhile, sites that allow everything hand their content to training datasets with no attribution, no backlinks, and no referral traffic in return. The same research found AI-referred traffic converts about 4.4 times better than standard organic search, which makes visibility in AI search results worth protecting, not blocking by accident.

How to control them, and what robots.txt cannot do

The first layer is robots.txt. One group per user-agent token is the clearest form: a Disallow rule for GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, and Meta-ExternalAgent blocks the training crawlers, while leaving OAI-SearchBot, Claude-SearchBot, and PerplexityBot alone. That file is a request, not a lock. RFC 9309, which standardized robots.txt in September 2022, states that its rules are not a form of access authorization, so a compliant crawler reads it and stays out while nothing technical stops one that does not. The second layer is enforcement at the server or CDN: return a 403 to the training user agents, which works on anything that identifies itself, compliant or not. The third layer is verification, because a user-agent string is text anyone can send. Check the source IP against the vendor's published list or its reverse DNS before trusting that a request claiming to be GPTBot actually is. A scraper wearing a famous bot's name is the oldest trick in this category, and the log line that says GPTBot is not proof until the IP agrees. If your traffic problem is the older kind of bot, how sites actually spot a headless browser covers the signals that give automation away, and why IP reputation stopped working explains why the address alone no longer settles it.