Can robots.txt stop malicious crawlers? Not that simple

What robots.txt actually does and where it fails. Malicious crawler fingerprints, Nginx UA blacklist and rate-limit configs, how to avoid false-positives when banning UAs, and incident response when a crawl takes the site down. FAQ.

Many operators assume a well-written robots.txt stops crawlers. It is a gentleman's agreement. This FAQ covers what robots.txt actually does, whether malicious crawlers honor it, whether it is still worth configuring, practical Nginx configs to intercept malicious crawlers, and incident response when a crawl knocks the site over.

Q: What is robots.txt, and what does it do?

robots.txt is a text file at the site root. User-agent and Disallow directives tell crawlers which directories they may fetch and which they may not. Key: it is advice, not enforcement. The protocol has zero technical binding force. Legitimate search-engine crawlers (Baidu, Google) honor it:

`User-agent: Baiduspider
Disallow: /admin/
Disallow: /member/

User-agent: *
Disallow: /private/

Sitemap: https://example.com/sitemap.xml`

Q: Do malicious crawlers honor robots.txt?

No. Their goal is the content you are trying to protect. They ignore the protocol, or invert it — you Disallow /admin/, they now know a high-value path exists. Using robots.txt against malicious crawlers is leaving the house key under the doormat.

Q: Is robots.txt still worth configuring?

Yes, but not as an attack control. Use it for SEO: steer legitimate spiders onto pages that should be crawled, conserve crawl budget, keep admin paths and duplicate pages out of the index. Malicious-crawler defense belongs one layer down.

Q: What fingerprints identify a malicious crawler?

• Rate: single-IP request rate far above human; 7×24 nonstop
• UA: empty UA; python-requests, Go-http-client, Scrapy and other client UAs; misspelled or stale browser UAs
• Behavior: HTML only, no JS/CSS/images; regular directory-traversal paths
• TLS fingerprint: JA3 does not match a real browser. A convincing UA still cannot hide the handshake.

Q: How does Nginx intercept malicious crawlers?

Step 1 — UA blacklist for obvious malicious clients:

`map $http_user_agent $bad_bot {
    default 0;
    ~*python-requests 1;
    ~*Go-http-client  1;
    ~*scrapy          1;
    ~*curl            1;
    ~*wget            1;
    ""                1;
}
server {
    if ($bad_bot) { return 403; }
}`

Step 2 — rate-limit whatever still gets through:

`limit_req_zone $binary_remote_addr zone=antispider:10m rate=10r/s;

location / {
    limit_req zone=antispider burst=20 nodelay;
}`

Tune rate to the actual workload. Start loose, then tighten. Avoid false-positives on real users and legitimate spiders.

Q: Will blocking empty UAs and script UAs false-positive?

Most production traffic can be blocked. Two exceptions: synthetic monitoring probes and compliant API clients. Safer path: log first, do not intercept — swap return 403 for logging, watch a week, enable the ban once you confirm no legitimate sources.

Q: The crawler took the site down. Now what?

• Parse access logs for source and path; single IP vs distributed crawl
• Single-IP high rate: ban the IP or prefix. Distributed low rate: CAPTCHA or JS challenge
• Nginx rate-limit as backstop; keep the database and upstream app alive first
• Longer term: WAF or CDN. Identify crawlers at the edge so origin never sees the traffic

Q: Any extra advice?

robots.txt, UA blacklist, rate-limit, WAF are stacked: robots.txt handles SEO allocation, Nginx handles obvious malicious traffic, WAF handles well-disguised crawlers. Do not bet on any single layer. Periodically verify traffic claiming to be Baiduspider/Googlebot (reverse-check IP ownership). Plenty of malicious crawlers impersonate search engines to scrape for free; fake spiders in logs often outnumber what you expect.