/PLAYBOOK · 2026-07-28

Malicious Bot Detection and Blocking Hands-on Configuration Guide

Four-layer bot defense at Nginx/CDN: User-Agent filtering, rate limiting, GeoIP blocking, and JS Challenge. Targets scrapers, credential stuffing, and content theft.

Scope

Malicious bots (crawlers) are a common traffic threat: content scraping, credential stuffing, CC attacks, fake signups, comment spam. Imperva: nearly 50% of 2025 global internet traffic was automated; about 30% of that was malicious. For site operators, security engineers, and CDN admins detecting and blocking common malicious bot traffic at the Nginx/CDN layer.

Prerequisites

Nginx web server (1.18+)
root or sudo
Basic Nginx config syntax
If using a CDN (Cloudflare, Alibaba Cloud CDN, etc.), CDN console access

Mechanism

Core idea: multi-dimension detection on behavior and request features. Typical dimensions: User-Agent (bots mimicking browsers with abnormal headers), request rate and interval (bot intervals are often highly uniform), path patterns (bots probing sensitive paths), IP reputation (known proxies or datacenter IPs), JA3 fingerprint (TLS handshake fingerprint; hard for bots to mimic a real browser). This tutorial implements Nginx-layer User-Agent filtering, rate limiting, and GeoIP blocking.

Procedure

Step 1: Configure a User-Agent blacklist

Create UA filter rules in the Nginx http block. Block known malicious Bot User-Agents:

`# Add inside the http block in /etc/nginx/nginx.conf

# Malicious UA regex map
map $http_user_agent $bad_bot {
    default 0;
    # Known malicious crawlers
    ~*curl 1;                           # curl-like bots (review: may block legitimate curl)
    ~*wget 1;
    ~*python-requests 1;
    ~*go-http-client 1;
    ~*scrapy 1;
    ~*java/ 1;
    ~*libwww-perl 1;
    ~*masscan 1;
    ~*nikto 1;
    ~*sqlmap 1;
    ~*nmap 1;
    ~*zgrab 1;
    ~*(data_fetcher|dataforseo|zoominfo) 1;
    ~*(semrush|ahrefsbot|majestic) 0;    # SEO tools; allow if needed
    ~*(mj12bot|dotbot|blexbot) 0;        # search-engine bots; usually allow
}

server {
    # Use in the server block
    if ($bad_bot) {
        return 403;
    }
}`

Step 2: Create a dedicated anti-bot config file

Extract anti-bot rules into a standalone file for management and reuse:

`# Create the anti-bot config
sudo vim /etc/nginx/conf.d/antibot.conf

# Fill in the following
# === Malicious bot identification and blocking rules ===

# 1. UA blacklist
map $http_user_agent $blocked_ua {
    default 0;
    ~*(masscan|nikto|sqlmap|nmap|zgrab|python-requests|go-http-client|scrapy|java/) 1;
}

# 2. Empty UA blocking (browsers always send a User-Agent)
map $http_user_agent $empty_ua {
    default 0;
    "" 1;
}

# 3. Restrict request methods (allow only GET/POST/HEAD)
if ($request_method !~ ^(GET|HEAD|POST)$) {
    return 405;
}

# Include this file in nginx.conf
# include /etc/nginx/conf.d/antibot.conf;`

Step 3: Configure rate limiting

IP- and URI-based rate limits are effective against crawler bots:

`# Add rate-limit zones in the http block of nginx.conf

# Per-IP limit: max 30 requests per second
limit_req_zone $binary_remote_addr zone=bot_limit:10m rate=30r/s;

# Combined IP + URI limit: for sensitive paths
limit_req_zone $binary_remote_addr$uri zone=sensitive_limit:10m rate=5r/s;

# Rate-limit login/registration endpoints
limit_req_zone $binary_remote_addr zone=auth_limit:10m rate=3r/m;

server {
    # Apply rate limiting (use inside location)
    location / {
        limit_req zone=bot_limit burst=50 nodelay;
    }

    # Protect API endpoints
    location /api/ {
        limit_req zone=sensitive_limit burst=10 nodelay;
    }

    # Protect login endpoints
    location /wp-login.php {
        limit_req zone=auth_limit burst=5;
    }
    location /wp-admin {
        limit_req zone=auth_limit burst=5;
    }
}`

Step 4: Configure GeoIP regional blocking

If the service is China-mainland only, block overseas IPs to cut the malicious bot attack surface:

`# Install the GeoIP2 module
# Ubuntu:
sudo apt install libnginx-mod-http-geoip2

# Download the GeoIP2 database (free edition)
sudo mkdir -p /etc/nginx/geoip
# Download from https://dev.maxmind.com/geoip/geolite2-free-geolocation-data
# or use a China-based mirror

# Configure in nginx.conf
geoip2 /etc/nginx/geoip/GeoLite2-Country.mmdb {
    $geoip2_data_country_code source=$remote_addr country iso_code;
}

map $geoip2_data_country_code $blocked_country {
    default 1;        # Block by default
    CN 0;             # Allow China
    HK 0;             # Allow Hong Kong (optional)
    TW 0;             # Allow Taiwan (optional)
    MO 0;             # Allow Macau (optional)
}

server {
    if ($blocked_country) {
        return 403;
    }
}`

Step 5: Configure JS Challenge (advanced bot detection)

For smarter bots that UA and rate limits miss, use a JS Challenge. Nginx has no built-in JS Challenge; combine with a third-party:

`# Option 1: Use Cloudflare JS Challenge
# Configure WAF rules in the Cloudflare dashboard
# Security → WAF → Create Rule
# Condition: Threat Score > 0 or Bot Score. After configuration, verify bot blocking as follows:

Verification 1: Simulate a malicious bot # Access the site with python-requests (should be blocked) curl -A "python-requests/2.28" https://helixcyber.ai/ # Expected: 403 Forbidden

Verification 2: Empty UA blocking curl -A "" https://helixcyber.ai/ # Expected: 403 or the default empty-UA deny page

Verification 3: Rate-limit test # Use a for loop to simulate rapid requests for i in {1..60}; do curl -s -o /dev/null -w "%{http_code}\n" https://helixcyber.ai/; done | sort | uniq -c # Expected: 503 (Service Temporarily Unavailable) after exceeding 30r/s

Verification 4: Inspect real blocking logs sudo tail -100 /var/log/nginx/access.log | grep "403" # Confirm the malicious bot IP and UA are logged correctly

FAQ

FAQ1: User-Agent filtering blocked legitimate search engines (Baidu, Google). How to fix?

In the map block, set known search-engine Bot matches to 0. Common ones: Baiduspider, Googlebot, Bingbot, Sogou, YandexBot. Verify with `nslookup` reverse DNS on the search-engine IP. Combine `$blocked_ua` with a search-engine IP-range allowlist.

FAQ2: Rate limiting returns 503 to real users. How to tune?

Check whether burst is too small. burst is the queue of requests allowed to temporarily exceed the limit. Use burst=50 nodelay on public pages such as the homepage; burst=10-20 on API. Raise the cap for logged-in users via cookie so humans and bots are not treated the same.

Summary

Malicious bot detection and blocking is baseline site security ops. This tutorial builds four layers: User-Agent filtering, rate limiting, GeoIP blocking, and JS Challenge. In production, pair CDN WAF (Cloudflare or Alibaba Cloud WAF) with local Nginx rules: CDN drops known bot IPs and high-volume attacks; Nginx adds behavior analysis and rate limits. Keep a regular bot-behavior review from access logs and tighten rules from that.