/PLAYBOOK · 2026-08-01

Nginx UA blacklist: block malicious crawlers and scanners

Block malicious User-Agents with Nginx map+if. Covers sqlmap, crawlers, and scanners. Hits return 403 without consuming app resources. Full config, verify commands, and a rate-limit fallback.

Scope

Scanners, crawlers, and scripts hit the site. Logs fill with high-volume User-Agents such as curl, sqlmap, python-requests. They consume bandwidth and app resources. Often they precede vuln probing. Block known-bad User-Agents at Nginx. Deploy on public-facing Nginx, reverse proxies, or load balancers. Fits WordPress and corporate sites as an access-layer control.

Prerequisites

Linux server (Ubuntu/Debian in this article). Nginx 1.18 or later.
Write access to /etc/nginx/. You can run nginx -t and systemctl reload nginx.
A working server block (e.g. /etc/nginx/sites-available/site.conf).

How it works

Match User-Agent against known malicious tools. A hit returns 403. Nginx maps UA to a flag with map in the http context, then if in server/location blocks the request. Nginx-layer blocking happens at the edge. No app-process cost. Regex, case-insensitive. One config, global effect.

UA is spoofable. A blacklist raises attacker cost. Combine with rate limits, WAF, and log tracing. Do not treat it as the only control.

Steps

1. Create a standalone UA blacklist config

Keep maps in their own file:

sudo mkdir -p /etc/nginx/conf.d
sudo nano /etc/nginx/conf.d/block_ua.conf

Write this (grouped by tool type, all ~* case-insensitive):

# Common CLI tools and downloaders
map $http_user_agent $bad_bot {
    default 0;
    ~*curl 1;
    ~*wget 1;
    ~*libwww-perl 1;
    ~*Go-http-client 1;
    ~*Java/ 1;
    ~*okhttp 1;

    # Vulnerability scanners
    ~*sqlmap 1;
    ~*nikto 1;
    ~*nmap 1;
    ~*masscan 1;
    ~*zgrab 1;
    ~*acunetix 1;
    ~*nessus 1;
    ~*fimap 1;

    # Crawlers and scrapers
    ~*scrapy 1;
    ~*python-requests 1;
    ~*aiohttp 1;
    ~*urllib 1;
    ~*HTTrack 1;
    ~*offline\ explorer 1;
    ~*semrushbot 1;

    # Directory brute-force and exploit tools
    ~*dirbuster 1;
    ~*dirb 1;
    ~*gobuster 1;
    ~*wpscan 1;
}

2. Include the map in the main config

Edit /etc/nginx/nginx.conf. Inside http { }, before any server block:

http {
    include /etc/nginx/conf.d/block_ua.conf;
    # ... rest of config
}

3. Enable blocking in the server block

Edit the site config. Add at the top of the server block:

server {
    listen 443 ssl;
    server_name helixcyber.ai;

    # UA blacklist: 403 on hit
    if ($bad_bot) {
        return 403;
    }

    # other config...
}

4. Rate-limit fallback for allowed tools (optional)

If a UA is too generic (e.g. python-requests), rate-limit instead of 403:

map $http_user_agent $suspicious_bot {
    default 0;
    ~*(python-requests|aiohttp|curl) 1;
}

# inside server:
limit_req_zone $binary_remote_addr zone=bot_limit:10m rate=5r/m;
server {
    if ($suspicious_bot) {
        limit_req zone=bot_limit burst=10 nodelay;
    }
}

Note: limit_req_zone belongs in http. limit_req is safer inside location. Putting it in a server if is demo only.

Verify

# 1. Syntax check
sudo nginx -t

# 2. Reload config
sudo systemctl reload nginx

# 3. Simulate a malicious UA; expect 403
curl -A "sqlmap/1.8.2" -I https://helixcyber.ai/
# Expected: HTTP/1.1 403 Forbidden

# 4. Simulate a normal browser UA; expect 200
curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/126.0" -I https://helixcyber.ai/
# Expected: HTTP/1.1 200 OK

Advanced check: run a real scanner and confirm 403 in access.log:

grep "403" /var/log/nginx/access.log | tail -n 20

FAQ

FAQ 1: Why do my own curl tests fail after I block curl?

Expected. The blacklist hits your tests too. Whitelist ops hosts, or test with a browser / custom UA, e.g. curl -A "Mozilla/5.0 Test" https://helixcyber.ai/. In production, do not blanket-block curl and wget. Rate-limit instead of 403 if needed.

FAQ 2: Attackers spoof UA. Is the blacklist still useful?

Yes. Most bulk scans keep default UAs. A blacklist cuts ~60%-80% of spray traffic. For spoofed UA, add limit_req, behavior analysis (rate, path patterns), and WAF rules. UA blacklist is layer one, not the only layer.

Takeaway

map + if blocks bad UAs at Nginx. Separate file for maintenance. Regex covers common scanners and crawlers. 403 at the edge, no app cost. Rate-limit as fallback. Key points: map in http, before server; match case-insensitive; nginx -t then reload. Combine UA blacklist, rate limits, and WAF: UA block + rate limit + behavior analysis. Keep access.log for tracing.