Site Crashed by Crawlers? Rate-Limit and Ban Playbook

Site suddenly slow, CPU spiked, bandwidth saturated. Often not an attack—malicious crawlers scraping nonstop. Eight Q&As on how to tell crawler-induced outages, hunt crawler IPs in logs, Nginx rate-limit and ban configs, and a long-term defense plan.

Site suddenly slow. CPU and bandwidth spike. First instinct is attack. After triage it is not DDoS: malicious crawlers scraping around the clock, exhausting resources. Crawler incidents and attack incidents need different playbooks: attacks you ban, crawlers you rate-limit. Eight Q&As on the full handling path after crawlers crash a site, from log hunt to Nginx rate-limit and ban configs.

Q: How do you tell a crawler crash from DDoS?

Read access-log patterns first. DDoS: high volume, usually monotonous (many identical URLs or non-page requests). Crawler scrape: requests walk page links. Many distinct URLs, relatively concentrated source IPs, regular intervals. Top talkers:

awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -20

One or a few IPs at tens of thousands to hundreds of thousands of requests/day: crawler. Sample their pages:

grep "1.2.3.4" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head

Distinct URLs, traversed in link order: classic collector crawler.

Q: Crawler IPs confirmed. Emergency first step?

Ban IPs to stop the bleed. Single IP via Nginx deny:

# inside the http or server block of nginx.conf
deny 1.2.3.4;
deny 5.6.7.0/24;

Then nginx -s reload. If crawler IPs are scattered and numerous, ban in iptables at the OS layer. Cheaper than Nginx:

iptables -I INPUT -s 1.2.3.4 -j DROP
iptables -I INPUT -s 5.6.7.0/24 -j DROP
iptables-save > /etc/sysconfig/iptables   # persist (CentOS)

Ban is emergency only. If the crawler rotates IPs, IP bans only buy time. Must pair with rate limits below.

Q: How do you configure Nginx rate limiting against crawlers?

This is the durable core. Define a limit zone in the http block, apply it in server or location:

http {
    limit_req_zone $binary_remote_addr zone=perip:10m rate=10r/m;
    limit_req_status 429;

    server {
        location / {
            limit_req zone=perip burst=20 nodelay;
        }
    }
}

Parameters: rate=10r/m means 10 requests/min per IP. Fine for normal users; collector crawlers get throttled hard. burst=20 allows 20 queued burst requests. nodelay serves burst immediately with no wait, so real users do not feel it. Throttled requests return 429. If crawlers hammer dynamic pages, tighten PHP/API locations only and loosen static, to cut collateral.

Q: Ban common malicious crawler User-Agents?

Yes. Zero-cost extra. Collector tools and scanners usually have distinctive UAs. Reject in Nginx:

if ($http_user_agent ~* (python-requests|scrapy|curl|wget|java|go-http-client|semrush|ahrefs|mj12bot)) {
    return 403;
}

Two notes: do not kill Baiduspider, Googlebot, and other search spiders—that hits indexing. UA is spoofable; serious malicious crawlers fake browser UAs. UA blacklist stops low-bar tools, not targeted collection. Pair with rate limits.

Q: Does robots.txt help?

Against legitimate crawlers, yes. Against malicious crawlers, no. Search spiders and spec-compliant collectors read robots.txt and honor it. Malicious crawlers and competitor collectors ignore it—or treat robots.txt as a map of valuable content. robots.txt is a policy statement, not a control: tell good crawlers what to skip. Stop bad crawlers with rate limits, bans, and WAF.

Q: Crawler spoofs a search-engine spider to bypass rate limits?

Verify first, then decide whether to allow. A real spider's IP reverse-resolves to the engine's domain. Baidu example:

host 116.179.32.1
# only a real spider if the result contains baiduspider/baidu.com

Stricter: forward + reverse DNS dual check. Confirmed fake spider: blacklist the real IP. If you will not verify yourself, treat claimed-spider requests whose IP does not reverse-resolve as ordinary visitors: rate-limit, no exemption.

Q: IP bans and rate limits in place, crawler still rotates IPs?

Targeted collection. Adversarial crawler. Manual IP bans cannot keep up with rotation. Escalate:

• CDN / WAF: filter requests first. Crawler fingerprints and anomalous rates die at the edge. Origin only sees clean traffic;
• JS challenge: real browsers execute JS; simple scrape scripts do not. Drops most amateur collectors;
• Lower scrape value: statically cache hot list pages. Crawler hits CDN cache, never origin DB.

For content you must protect from collection (quotes, data lists), tech alone rarely fully stops it. Pair with login-gated views and dynamic loading to raise collection cost.

Q: Additional notes?

Full loop: log-hunt anomalous IPs → iptables/Nginx ban to stop bleed → global per-IP rate limit + 429 as standing control → UA blacklist as zero-cost extra → CDN/WAF + JS challenge for adversarial crawlers. Add a simple monitor: daily Top 10 IPs by request count, alert past a threshold. More proactive than waiting for the site to stall. Crawler defense is not a one-shot ban. Make scrape cost exceed value. They leave.