How to Catch Crawlers Posing as Real Users: Behavioral Detection in Practice

Modern scrapers forge UA, rotate proxies, and throttle. IP and User-Agent alone no longer separate them. Eight Q&As cover detecting disguised crawlers via request cadence, path patterns, and resource loading, with Nginx log fields, awk analysis commands, and a graded response playbook.

Many operators find crawlers keep scraping after IP and User-Agent bans — collectors now rotate proxy IPs, forge browser UAs, even mimic click intervals. Catching crawlers that pose as real users requires behavioral features. Eight Q&As cover the full practical method.

Q: Why do crawlers impersonate real users?

Because the simplest blocks are identity-based, and changing identity bypasses them: ban IP, switch proxy pool; ban UA, copy a Chrome UA string; rate-limit, slow down until a single IP looks normal. Identity is forgeable. Behavioral regularity is not — humans jump, randomize, skip; programs traverse, uniform, complete. Behavior detection targets that gap.

Q: Which dimensions separate humans from disguised crawlers?

Three. Hit two or more and you can usually confirm:

• Request cadence: human intervals vary long/short and do not stay uniform at 03:00. Crawlers are highly uniform (fixed 300ms or 1s) or drop the interval and burst concurrently.
• Path pattern: humans enter from a landing page, browse popular content, leave. Crawlers walk IDs in order, increment page params continuously, or hit large volumes of detail pages without ever touching the homepage.
• Resource loading: humans load CSS/JS/images/fonts. Pure HTTP collectors usually take only HTML and APIs. Logs show .html only, no static assets.

Never loading static assets is the hardest evidence of the three, and the easiest to see in logs.

Q: Which log fields do you need for behavior detection?

Default logs have IP, time, request, UA, status, size — but not request interval or response time, which makes analysis painful. Add these two fields in Nginx:

`log_format behavior '$remote_addr | $time_iso8601 | $request | '
                    '$status | $body_bytes_sent | '
                    '$request_time | $upstream_response_time | '
                    '\"$http_user_agent\" | \"$http_referer\"';
access_log /var/log/nginx/access.log behavior;`

Field notes: $request_time is this request's duration; scraped pages tend to be slow. $upstream_response_time is backend processing time; shows whether the DB is being hit. Empty $http_referer means a direct URL; humans navigating in-site usually carry a referrer. Apply with nginx -s reload.

Q: How do you quickly find crawlers with overly uniform cadence?

Count requests per IP, take the head of the list:

`awk -F' \\\\ | ' '{print $1}' access.log | sort | uniq -c | sort -rn | head -20`

Then inspect one suspect IP's timestamps for uniform gaps:

`grep \"1.2.3.4\" access.log | awk -F' \\\\ | ' '{print $2}' | head -40`

If second-level timestamp deltas are nearly constant (always 0.3 or 1.0 s), that is classic programmatic throttle. Human deltas are messy. Add a static-resource check:

`# check whether this IP has requested any static assets
grep \"1.2.3.4\" access.log | grep -cE '\\\\.(css|js|png|jpg|gif)'
# result of 0 or extremely low: almost certainly a collector`

Q: Can User-Agent and Referer still help?

Yes, as supporting evidence only. Do not conclude from them alone. Three anomalies in practice:

• UA vs behavior mismatch: claims Chrome, never loads CSS/JS, never sends Referer.
• Stale or anomalous UA: mass traffic on a years-old UA, or UA containing python-requests, Go-http-client, curl, etc.
• Global UA uniformity: hundreds of thousands of requests, one UA, and that UA's share does not match real browser version mix.

Once confirmed, stop the bleed with a blacklist (put it at the top of the server block; match-and-deny is cheapest):

`if ($http_user_agent ~* \"(python-requests|Go-http-client|curl|wget|scrapy|java/)\") {
    return 403;
}
# or ban the CIDR directly (use a real prefix; do not copy this example)
deny 1.2.3.0/24;`

Note: if in Nginx is inefficient. High-traffic sites should pre-decide with map or geo, or put the rule on the CDN/WAF, so every request does not run a regex.

Q: Once identified, how do you respond?

Do not 403 everything. Graded response is kinder to SEO and the business:

• L1 (suspect): log only, no block. Tag the IP onto a watchlist for trend analysis.
• L2 (light): rate-limit. limit_req crushes that source to a crawl:

`limit_req_zone $binary_remote_addr zone=botzone:10m rate=5r/m;
location / {
    limit_req zone=botzone burst=10 nodelay;
    limit_req_status 429;
}`

• L3 (confirmed collector): return fake data or empty pages so the scrape is worthless; put core content behind login.
• L4 (malicious high intensity): ban IP / CIDR at the edge so origin resources are not burned.

The point of grading: you are stopping bulk, high-frequency, full-corpus collection, not occasional one-off fetches. Over-blocking raises your own ops cost.

Q: Can behavior alone catch collectors spoofing Baidu spider?

Behavior helps; identity verification finishes it. Filter suspects by behavior, then reverse-check identity: real search-engine spiders reverse-DNS to an official hostname (e.g. baiduspider's official domain) and forward-resolve back to the same IP. Spoofers typically only change the UA string; reverse DNS will not match. Example:

`# reverse-DNS to hostname
dig -x 1.2.3.4 +short
# then forward-resolve that hostname; check it maps back to the same IP
dig +short <hostname from previous step>`

Both directions must match before you whitelist as a real spider. Mismatch: treat as a normal crawler even if UA is identical. Re-run verification in batch on a schedule; spoofed UAs keep changing.

Q: Any extra advice?

Three lessons. First, quantify before you act: banning IPs without reading logs hits real users and search engines. Run a week of stats — Top IPs, UA mix, static-resource request rate — then write rules. Second, push blocking to the edge: origin-side behavior detection costs server CPU; CDN/WAF nodes can do rate, fingerprint, and behavior at the edge so origin only sees clean traffic, and scraping does not hurt real visitors. Third, accept imperfect coverage: collectors with proxy pools and fingerprint pools are hard to stop completely. A sane goal is make collection slower, more expensive, and incomplete — not 100% block. Permission-tier core data and cache hot pages usually beat more complex rules.