← News
/PLAYBOOK · 2026-09-05

How do you identify crawlers spoofing Baiduspider? Verification methods included

Fake Baiduspider crawlers bypass anti-bot, burn bandwidth, scrape content. Real Baiduspider UA and IP traits, reverse-DNS verification script, Nginx real-spider allowlist, and iptables bans for fakes.

Q: Why specifically identify fake Baiduspider?

Malicious crawlers prefer search-engine spider disguises because many sites allowlist search-engine IPs: no rate limit, no CAPTCHA. Spoofing Baiduspider is obvious: bypass anti-crawl, high-rate fetch without intercept, burn server resources. Typical damage: scrape original content, high-rate requests knock the server over, use the crawler as cover while probing vulns.

Q: What does a real Baiduspider UA look like?

PC UA format:

`Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)`

Mobile includes Baiduspider-mobile or Mobile. But UA is a client self-declaration and can be forged at will. Anyone can change UA to Baiduspider with one curl. UA-only authenticity checks are unreliable. Combine with source-IP verification.

Q: How does reverse-DNS verification work?

Baidu's official recommended method. Two steps:

`# Reverse DNS lookup (replace IP with the actual source IP)
host 116.179.32.1
# Example output: ... domain name pointer crawl-xxx.baidu.com

# Forward verification: confirm it resolves back to the same IP
host crawl-xxx.baidu.com`

Both steps must pass. PTR without forward verification can be fooled by a forged PTR.

Q: How do you batch-verify spider IPs in logs?

Filter all source IPs claiming Baiduspider from access.log, then verify each:

`grep Baiduspider access.log | awk '{print $1}' | sort -u | while read ip; do
  host $ip 2>/dev/null | grep -q 'baidu\\.com' || echo \"suspicious fake spider: $ip\"
done`

Script logic: any IP whose UA contains Baiduspider but whose PTR does not land on a baidu.com hostname is marked suspect. Dump suspect IPs, then spot-check.

Q: Can JA3/TLS fingerprints identify fake spiders?

Yes, and this layer is harder to forge than UA. Real spiders are launched by a fixed HTTP client on Baidu servers; TLS handshake parameters (cipher suites, extension order, etc.) produce a relatively stable JA3. Attackers can fake UA; their TLS stack still does not match the official crawler. Collection tools and scripting languages (Python requests, etc.) have JA3 fingerprints clearly distinct from the real spider. Whitelist JA3 at WAF or an Nginx module. See JA3 fingerprint identification in practice for the internals.

Q: How does Nginx allowlist only real spiders?

If UA claims Baiduspider but source IP is not in official ranges, 403. Implement with geo + map:

`geo $baidu_ip {
    default 0;
    # Example CIDRs; use the latest IP ranges published on Baidu Webmaster Tools
    116.179.32.0/24  1;
    116.179.33.0/24  1;
}

.map \"$http_user_agent:$baidu_ip\" $fake_spider {
    # UA claims Baiduspider but IP is not in the allowlisted ranges → treat as fake spider
    \"~*Baiduspider:0\"  1;
    default            0;
}

server {
    if ($fake_spider) {
        return 403;
    }
}`

Two points: CIDRs in the geo block must be updated regularly against the latest ranges published on Baidu Webmaster; a wrong range blocks real spiders. if is used only with return — the safe if usage in Nginx, no config-trap.

Q: After you confirm a fake spider, what then?

Q: Any extra advice?