TCP SYN Flood Defense in Practice: Kernel Parameters and Firewall Configuration

SYN Flood exhausts the half-open queue and causes denial of service. This article gives Linux kernel parameters, iptables rate limits, Nginx handshake tuning, and upstream L4 anti-DDoS configs. Copy-paste ready, with verification and FAQ.

SYN Flood is a classic DDoS: the attacker sends a flood of SYNs with spoofed sources and never completes the handshake, filling the server half-open queue until new connections fail. This article covers Linux kernel parameters, iptables, Nginx, and upstream L4 anti-DDoS for SYN Flood defense.

Scope

• Large numbers of SYN_RECV connections, application stalls, or a port that stops answering
• First-response self-help on a public host under protocol DDoS
• Baseline hardening before a new server goes live

Prerequisites

• Linux kernel 3.10+ (parameters apply to CentOS/Debian/Ubuntu)
• root
• Nginx 1.12+ (application-layer section optional)
• On cloud hosts, confirm the vendor baseline DDoS protection and anti-DDoS product

SYN Flood mechanics

In the TCP handshake the server allocates half-open state on SYN, replies SYN-ACK, and waits for ACK. The attacker spoofs sources, never ACKs, and fills the queue so legitimate SYNs cannot land. Defense: shorten half-open timeout, enable SYN Cookie for stateless replies, filter spoofed traffic at the edge.

Procedure

Step 1: Harden kernel parameters (primary control)

# /etc/sysctl.d/99-anti-synflood.conf
# Enable SYN Cookie: no half-open queue use; stateless replies
net.ipv4.tcp_syncookies = 1
# Enlarge SYN backlog (default 128 is far too small)
net.ipv4.tcp_max_syn_backlog = 65536
# Cut SYN-ACK retries from 5 to 1; free spoofed state faster
net.ipv4.tcp_synack_retries = 1
# Enable TIME_WAIT reuse
net.ipv4.tcp_tw_reuse = 1
# Accept queue and local port range
net.core.somaxconn = 65536
net.ipv4.ip_local_port_range = 1024 65000
# conntrack table ceiling
net.netfilter.nf_conntrack_max = 1048576

sysctl -p /etc/sysctl.d/99-anti-synflood.conf
sysctl net.ipv4.tcp_syncookies   # output = 1 means applied

Step 2: Rate-limit new connections with iptables

# Drop a source with more than 50 concurrent connections
iptables -A INPUT -p tcp --syn --dport 443 -m connlimit --connlimit-above 50 -j DROP
# Drop a source exceeding 10 new connections per second
iptables -A INPUT -p tcp --syn --dport 443 -m hashlimit \
  --hashlimit-name syn_flood --hashlimit-above 10/sec \
  --hashlimit-burst 20 --hashlimit-mode srcip -j DROP

# Drop illegal TCP flag combinations (spoof signatures)
iptables -A INPUT -p tcp --tcp-flags ALL NONE -j DROP
iptables -A INPUT -p tcp --tcp-flags SYN,FIN SYN,FIN -j DROP
iptables -A INPUT -p tcp --tcp-flags SYN,RST SYN,RST -j DROP

iptables-save > /etc/iptables/rules.v4

Step 3: Shrink handshake pressure in Nginx

# nginx.conf key parameters
worker_rlimit_nofile 100000;
events {
    worker_connections 50000;   # per-worker connection cap
}
http {
    # Cut occupancy time of dead connections
    client_header_timeout 10s;
    keepalive_timeout 30s;
}

nginx -t && systemctl reload nginx
# File descriptor ceiling (persist in /etc/security/limits.conf)
ulimit -n 100000

Step 4: Upstream anti-DDoS as last line

Host-only controls cannot absorb 10Gbps+ floods. Terminate attack traffic upstream:

• Enable vendor baseline DDoS protection; buy elastic anti-DDoS if the business requires it
• Point DNS to the anti-DDoS CNAME; origin security group allows only scrubbing-center return-to-origin IPs
• During an attack, confirm drop efficacy on the anti-DDoS console reports

Verification

# Half-open count: under attack it should not pin at the queue ceiling
ss -ant state syn-recv | wc -l

# Stack counters
netstat -s | grep -iE 'SYNCookie|SYNs to LISTEN'
# Rising passive cookie count after a flood means syncookies are working

cat /proc/sys/net/ipv4/tcp_syncookies   # expect 1

• Legitimate load-test connection success rate should stay above 99%
• Under a replayed flood the host should still accept SSH and serve core traffic

FAQ

FAQ1: tcp_syncookies is on, service is still down. Why?

SYN Cookie only stops half-open queue exhaustion. Saturated bandwidth or CPU still takes the service down. Check inbound volume on cloud metrics against the bandwidth cap. Pure volumetric floods above 1Gbps must go to upstream scrubbing. Host parameters only delay collapse; they do not fix a full pipe.

FAQ2: Will connlimit/hashlimit false-positive NAT users?

Yes. A company or campus sharing one egress IP can trip the limit. Fix: whitelist known large-customer egress IPs, and put those rules before the rate-limit rules. Raise hashlimit-above to 2× peak legitimate new connections per IP, then re-observe.

Summary

SYN Flood defense is layered: kernel parameters turn stateful wait into stateless reply; iptables drops spoofed and over-rate SYNs at ingress; Nginx cuts handshake occupancy; upstream anti-DDoS absorbs volumetric floods. All four together minimize business impact.