Why bots read your robots.txt before they attack
A file that asks nicely, and nothing more
Every bot that arrives at a store reads the same file first, or is supposed to. Robots.txt sits at the root of the site and tells crawlers which paths they may visit and which they should leave alone. It was proposed in 1994 by Martijn Koster and standardized as RFC 9309 in September 2022, and in all that time it has never become access control. It is a request. The bots that honor it do so voluntarily, and the bots that do not were never going to ask permission.
The file is everywhere. The HTTP Archive counts robots.txt on 94 percent of the 12 million sites it tracks. It is one of the most-read files on the internet, and most of what reads it is not a search engine. Price scrapers, inventory monitors, credential stuffing tools, and vulnerability scanners all start there, because it is free, it is public, and it answers the first question an attacker asks: what does this site think is worth hiding?
The Disallow list is a map
Read a store's robots.txt and you can see its floor plan. Disallow: /admin/ says where the admin panel lives. Disallow: /checkout/ says where the money moves. Disallow: /account/ says where the logins are. Disallow: /search?q= says the search page can be abused. Each line is a path the owner cared enough to name, and each named path is a path worth testing.
The honest version of this is that the file also tells the good bots where the waste is. A search crawler that obeys Disallow does not spend its crawl budget on your cart pages or your filtered search results, and your real pages get indexed faster. The file earns its keep with the good bots. The cost is that it also hands the bad ones a list, and there is no version of the file that does both jobs at once.
Who obeys it, and who never reads it
Search engines obey it, mostly, because their business depends on being allowed back. Price comparison tools and research crawlers usually obey it too, because they have a reputation to keep and a rate limit to respect. These are the bots the file was built for, and it works for them.
The bots that matter do not read it, or read it and ignore it. A credential stuffing tool does not care about your Disallow lines. A scraper with rotating residential proxies does not check the file before every request. The most telling sign is the bot that reads robots.txt and then immediately requests a Disallowed path: it has just told you it is not a search engine, and it has told you exactly what it wants.
What belongs in the file, and what does not
Keep the file short and boring. Disallow the paths that waste crawl budget: carts, checkout, account pages, filtered search, internal search results. Do not list paths that reveal something a human would not already know. If your admin panel lives at a secret path, do not write the secret path in a public file. The whole point of a secret path is that it is not published, and robots.txt publishes it.
And treat the file as a tripwire, not a wall. Log requests to robots.txt and watch what comes next. A client that fetches the file and then requests a Disallowed path is a bot worth blocking, and the request for the file itself is the earliest signal you will get. The file cannot stop an attack. It can tell you one is starting, and that is worth more than the crawl budget it saves.
Keep reading: Good bots vs bad bots: which crawlers to let through, Residential proxy networks: why the worst bot traffic has a home address, and Why rate limiting alone never stops a distributed bot attack cover the nearby cases in this series.
Frequently asked questions
Does robots.txt block bad bots from my site?
No. Robots.txt is a voluntary request, standardized as RFC 9309, with no enforcement mechanism. Well-behaved crawlers obey it. Malicious bots ignore it or use it as a list of interesting paths to test.
Why would a bot read robots.txt before attacking?
Because the Disallow lines name the paths the owner wants hidden, which are usually the admin panel, login, checkout, and account pages. To an attacker, that is a map of where the valuable functions live.
Should I list my admin path in robots.txt?
No. If the admin panel lives at a non-obvious path, writing it in a public file publishes it. Keep robots.txt to paths that waste crawl budget, like carts and filtered search, and protect the admin panel with authentication instead.
How can I tell a bad bot from a good one using robots.txt?
Watch the logs. A client that fetches robots.txt and then immediately requests a Disallowed path has told you it is not a search engine. That sequence is one of the earliest and most reliable bot signals a store has.