Good Bots vs Bad Bots: Which Crawlers to Let Through

The user agent string is a costume, not an ID

Any bot can claim to be Googlebot. The string is one line of text the client sends, and a scraper sends whatever string gets it through the door. The only check that means anything is reverse DNS: look up the IP address, confirm it resolves to a hostname owned by the crawler's operator, and confirm that hostname resolves back to the same IP. Google, Bing, and the other major crawlers all publish the ranges and hostnames to check against.

That check is the difference between a block list that works and one that blocks your own search traffic. A rule that bans a user agent string bans the honest crawler and misses the scraper wearing its name. A rule that verifies the IP keeps the honest one and catches the impostor, because the impostor cannot fake the DNS record.

The crawlers that pay their way

Search crawlers are the obvious keepers. Googlebot, Bingbot, and DuckDuckBot decide whether your product pages appear in search at all, and blocking one of them is the same as delisting yourself from that engine. They are also polite about it: they respect crawl delay signals, they come from published IP ranges, and they identify themselves consistently.

The quieter keepers are link previewers and uptime monitors. When someone shares your product on a messaging app or a social feed, the previewer fetches the page to build the card, and a blocked previewer means a blank share with no image and no price. Your uptime monitor hits the homepage every minute to confirm the store is alive, and blocking it means you find out about an outage from a customer instead of an alert.

The AI crawlers and the robots.txt question

A new category sits between good and bad: the AI training crawlers. GPTBot, CCBot, ClaudeBot, and Google-Extended all say they honor robots.txt, and that is the lever you have. Robots.txt is voluntary, it is a request, not a lock, and RFC 9309 standardised the format without giving it any enforcement power. A crawler that ignores it is not breaking a law, it is ignoring a request.

So the decision is a business one, not a technical one. If you want your product descriptions and guides out of training data, disallow the AI crawlers in robots.txt and check your logs to see whether they comply. If you want the traffic that comes from AI search answers, let Google-Extended through and block the rest. Either way, write the rule down, because the default is that everything is allowed.

Blocking the rest without blocking your customers

The bots to block are the ones that take value and return nothing: scrapers copying your catalogue and prices, credential stuffers testing stolen passwords at login, inventory hoarders holding stock in carts. They share a shape in the logs, high request rates, no asset loading, sessions that never touch the checkout the way a human does.

Start with behavior, not identity. A rule that blocks any client making fifty requests a minute catches the scraper and leaves the search crawler alone, because the search crawler never moves that fast on a small store. Then layer identity on top for the edge cases: verified crawlers get a pass, everything else gets the rate limit. That order matters, because identity alone is spoofable and behavior alone is noisy.

Keep reading: Headless Browsers vs Real Browsers: How Bot Detection Tells Them Apart, Why Real Users Fail CAPTCHA and Bots Pass It, Credential stuffing is not password spraying: why the defence differs.