Site data bulk-scraped by crawlers? Practical anti-collection

Product prices, article bodies, and user lists bulk-collected by peers are a loss many site ops never see. Eight Q&As cover how to confirm collection from logs, why IP bans fail against professional crawlers, how to deploy honeypot trap links and API signatures, and full Nginx rate-limit plus staged-verification configs.

Traffic is not high, the server is not saturated, but peers keep publishing your content first and the quote sheet walks off the whole site — that is usually a malicious crawler bulk-collecting data. Collection crawlers do not aim to knock the site down; they quietly empty the data. So the protection focus is not IP bans as bleed-stop, but raising collection cost. Eight Q&As cover a complete practical anti-collection scheme.

Q: How do you tell the site is being bulk-collected?

Three access-log signatures. First, a single IP's request volume far exceeds a normal user. A real visitor tops out at a few hundred requests a day; a collector doing tens of thousands a day is common:

awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -20

Second, URLs traverse in a pattern — sequential IDs, contiguous pagination params. Pull one suspect IP's URLs:

grep "1.2.3.4" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -30

Third, User-Agent is highly uniform: tens of thousands of requests, one UA, no CSS/JS/images, HTML and APIs only. Hit two of three and you can basically confirm a collection crawler.

Q: Why does IP ban alone fail to stop collection?

Professional collectors already use proxy IP pools. Ban one IP, they rotate the next. Manual IP bans cannot keep up. Ban is emergency bleed-stop, not a scheme. In an emergency, apply both layers; the system layer costs less CPU than the app layer:

# System-layer IP ban (lowest resource cost)
iptables -I INPUT -s 1.2.3.4 -j DROP
iptables -I INPUT -s 5.6.7.0/24 -j DROP

# Nginx-layer IP ban (nginx -s reload after the change)
deny 1.2.3.4;
deny 5.6.7.0/24;

The real fix is making IP rotation useless. The items below do that.

Q: How do you deploy honeypot trap links? Can they catch crawlers disguised as normal users?

Yes. One of the highest ROI controls. Principle is simple: plant a hidden link in the page — normal users never see it and never click it; crawlers parsing HTML follow it. Whoever hits that URL is a crawler:

<!-- Hidden from normal users; crawlers will follow and scrape it -->
<a href="/trap-honeypot-8f3a/" style="display:none" rel="nofollow">More content</a>

Matching Nginx rule: trap path is denied, IP is logged for later bulk ban:

location = /trap-honeypot-8f3a/ {
    access_log /var/log/nginx/honeypot.log;
    return 403;
}
location ~* ^/trap-honeypot- {
    return 403;
}

Advanced play is multi-stage traps: stage one is the hidden link; stage two is a link that only JS can generate (a real browser can click it; a bare HTTP client cannot assemble it). Chained, they filter out most mid-tier collectors.

Q: APIs are being scraped. How do dynamic tokens and signatures stop it?

APIs are the easiest surface to bulk-hit — no page-render tax. Core idea: every request carries a time-bounded signature so the collector cannot replay the URL as-is:

# Signature example (server-side verification)
sign = md5(uid + timestamp + nonce + secret)
# Server-side requirements:
# 1. timestamp must not differ from server time by more than 300 seconds
# 2. nonce must not repeat within 5 minutes (record with Redis SETNX)
# 3. sign must match, otherwise 403

Pair with Redis counters for rate limits, per user or per IP:

# At most 60 calls per uid per minute
INCR api:rate:{uid}:{minute}
EXPIRE api:rate:{uid}:{minute} 60
# Over 60, return 429 immediately

Signature answers whether they can replay. Rate limit answers how fast they can hit. Both must be on. Signature without rate limit: once they have a valid sign they still spray at high frequency.

Q: What are content masking and data watermarks for?

Post-facto tracing and value reduction. The former lets you prove they scraped you after a copy; the latter makes the stolen data worth less:

• Data watermark: inject tiny per-user diffs into visible content (invisible characters, slight order tweaks, amount decimal diffs). After they lift the whole site, the diffs identify the leak source.
• Content masking: render only necessary fields on the page. Full data (full phone numbers, full quote line items) requires login or application. Turns "scrape and get everything" into "scrape and get fragments".
• Image watermarks: images leave with a mark, lowering reuse intent.

Watermark value is deterrence and evidence. It is not an intercept control. Combined with the honeypot it forms a complete evidence chain.

Q: How do you configure Nginx-layer rate and connection limits?

This is the site-wide backstop. Every collector is constrained per IP or per connection. Define zones in the http block; apply in server/location:

http {
    # Per-IP request rate limit
    limit_req_zone $binary_remote_addr zone=anti_crawl:10m rate=20r/m;
    # Per-IP concurrent connection limit
    limit_conn_zone $binary_remote_addr zone=perconn:10m;
    limit_req_status 429;
    limit_conn_status 429;

    server {
        location / {
            limit_req zone=anti_crawl burst=30 nodelay;
            limit_conn perconn 15;
        }
        # Tighter limits on API paths
        location /api/ {
            limit_req zone=anti_crawl burst=10 nodelay;
        }
    }
}

Parameter meaning: rate=20r/m is 20 requests per IP per minute — enough for normal browsing, collectors get heavily throttled; burst=30 allows a burst queue; nodelay processes burst requests immediately so real users are not slowed; limit_conn 15 caps 15 concurrent connections per IP — specifically hits multi-thread collectors. Tighten API paths harder than pages: humans read pages slowly; API collectors spray at millisecond intervals.

Q: What about adversarial crawlers that rotate IPs and spoof UA?

At this point the opponent is targeted collection. Single-point controls are not enough. Escalate to staged verification:

• Level 1: JS challenge — a real browser executes JS and gets a pass cookie; a pure HTTP scrape script cannot. Drops most junior collectors once enabled.
• Level 2: behavior features — does the path inside a session match human patterns, are mouse/scroll events present, were CSS/JS loaded. Anomalies bounce.
• Level 3: stricter CAPTCHA — CAPTCHA high-frequency sources. Occasional real users can live with it; collector cost spikes.
• Level 4: edge intercept — put CDN/WAF in front, drop frequency-anomalous and fingerprint-suspect requests at the edge. Origin only receives clean traffic. Cache high-frequency pages so collection burns cache, not the database.

These four are stacked, not replacements: the first three answer whether you can identify; the fourth answers who pays the cost after identification.

Q: Any extra advice?

One line: the goal of anti-collection is not a clean ban list, it is making collection cost exceed return. Land in cost order, low to high: first inspect logs to confirm collection signatures (zero cost); add honeypot traps and global Nginx rate limits (low cost, fast effect); then API signatures + Redis rate limits (medium change volume); finally decide on CDN/WAF and staged verification by data value. Two long-running chores: run a Top IP stats script daily, alert if request volume suddenly 10x; permission-tier core data, expose as little as possible on the page. Data that is not exposed has no collection value.