How to Distinguish Malicious Crawlers from Search Engine Bots? Don't Hurt SEO

How to tell malicious crawlers from search engine bots. Covers identity, behavior, and protocol differences, reverse-DNS verification for real vs fake spiders, and Nginx configs to allow spiders while blocking bad bots—stop crawlers without killing SEO.

Many site operators cannot tell malicious crawlers from search engine bots. They either let bad crawlers crush the server, or blanket-block UAs and also stop Baidu and Google spiders—indexing falls off a cliff. This FAQ covers the differences, how to verify real vs fake spiders, Nginx allow/block configs, and how to avoid hurting SEO when blocking.

Q: What is the actual difference between malicious crawlers and search engine bots?

Both are automated programs. The difference is identity, purpose, and whether they follow the rules:

• Search engine crawlers: issued by Baidu, Google, Bing, etc. Fixed UA (e.g. Baiduspider, Googlebot). Honor robots.txt. Crawl frequency is quota-controlled. Purpose is building a search index.
• Malicious crawlers: purposes vary—bulk content scrape, credential stuffing/register, API data harvest, vuln scan, fake traffic. UA is forged at will. They do not read robots.txt. No frequency cap. Often rotate identity via proxy IP pools.

One-line summary: look at identity claim, crawl behavior, and protocol compliance together. Any single signal can mislead.

Q: How to make a first-pass judgment from User-Agent?

Mainstream search spider UAs have a fixed format. Count them in Nginx logs first:

`grep -E "Baiduspider|Googlebot|bingbot" /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20`

Legitimate spiders have a pattern: they hit robots.txt, sitemap, and in-page links first; URL distribution is spread out; frequency is steady. Malicious crawlers often hammer the same URL set, or walk a list in mechanical order. On frequency: Baidu official crawl has a pressure quota for small/medium sites. If a "spider" hits the same endpoint dozens of times per second, it is almost certainly fake.

Q: UA can be forged at will. How to verify real vs fake spiders?

Standard method is reverse DNS: reverse-lookup the request IP to a hostname, then forward-resolve that hostname back to an IP. Both sides must match for a real spider.

`# Assume the request IP is 116.179.32.18 (common Baidu spider netblock)
host 116.179.32.18
# Returns something like: baiduspider-116-179-32-18.crawl.baidu.com
host baiduspider-116-179-32-18.crawl.baidu.com
# The IP from forward resolution must match the request IP exactly; otherwise it is forged`

Googlebot maps to `.googlebot.com`, Bing to `.search.msn.com`. Daily ops does not need per-request verification. Collect official spider netblocks into an IP allowlist. Baidu publishes spider netblock notes on the webmaster platform (Baidu Search Resource Platform); download and check periodically.

Q: Can robots.txt stop malicious crawlers?

No. robots.txt is a gentlemen's agreement: real search engines honor it and crawl as declared. Malicious crawlers never read it—and may treat Disallow paths as a map of sensitive directories and crawl those harder. Still write robots.txt, but it only works on search engines. Malicious crawlers need technical blocks—rate limiting, UA denylist, frequency fingerprints, bot checks, in combination.

Q: How to allow search engines and block malicious crawlers in Nginx?

Two steps: map UA into three classes (spider, known-bad, other); rate-limit the other class; reject the bad class outright.

`map $http_user_agent $bot_class {
    default                     normal;
    ~*Baiduspider               spider;
    ~*Googlebot                 spider;
    ~*bingbot                   spider;
    ~*(python-requests|scrapy|HttpClient|Go-http-client|curl) badbot;
}
limit_req_zone $binary_remote_addr zone=bot_req:10m rate=5r/s;

server {
    location / {
        if ($bot_class = badbot) { return 403; }
        if ($bot_class = spider) { limit_req off; }
        limit_req zone=bot_req burst=10 nodelay;
        limit_req_status 429;
        ...;
    }
}`

Key points: spider class gets no rate limit, preserving search-engine crawl quota; badbot returns 403 immediately; normal UA falls through to rate=5r/s as a backstop. Tune thresholds after load-testing your own peak. Do not copy blindly. After edits, `nginx -t` then reload.

Q: What common malicious crawler types exist? How to block each?

• Scraper crawlers: lift the whole site. Trait: walk all list and detail pages. Block with rate limiting + frequency fingerprints.
• Credential-stuffing/register bots: only hit login, register, SMS endpoints. Trait: abnormal POST rate. Block with captcha + API signatures.
• Vuln scanners: awvs, sqlmap, dirsearch. Trait: mass 404 path probes, characteristic payloads. Block on UA/path signatures.
• Load-test/attack type: high-frequency hits on a single endpoint. Essentially CC. Block with edge WAF + rate limiting.

Different types, different policy. Confirm the type from logs first, then treat the cause. Better than a blind full block.

Q: How to avoid hurting SEO when blocking crawlers?

• Allowlist first: permit official spider netblocks, then apply block rules. Order cannot reverse.
• Leave headroom on thresholds: rate-limit threshold at least 3~5× normal user headroom. Watch logs 48 hours, then tighten gradually.
• Status codes matter: for suspected false positives return 429 or 503, not 403. Search spiders that see 429/503 slow down and retry; they do not drop the listing immediately.
• Watch indexing: after rules go live, watch Baidu Webmaster Platform crawl anomalies and index-volume curves for two weeks. If volume drops, roll back immediately.

Q: Any extra advice?

Distinguish and block is only step one. Long term, do three things: 1) script spider verification; periodically check that spider UAs in logs map to IPs in official netblocks, so fake spiders cannot lurk; 2) put bot checks or signatures on core APIs so crawlers can see data they cannot take; 3) if the site is chronically harassed by high-frequency crawlers and you have limited energy to tune Nginx, put a WAF with crawler recognition in front of the server (e.g. Baidu Cloud Protection). Hand UA fingerprints, frequency analysis, and JS challenges to cloud policy. Origin only handles good traffic. Cheaper than endlessly patching origin.