Linux ulimit Resource Limits in Practice: Hardening Against Resource-Exhaustion Attacks

Fork bombs and connection-exhaustion DoS let a runaway process take down the host. Full ulimit config: limits.conf global baseline, systemd per-service LimitNOFILE, Nginx worker_rlimit_nofile coupling. Includes verification commands and three failure-to-apply checks. For high-concurrency services and multi-user hosts.

Applicable Scenarios

ulimit resource limits stop a single process or user from exhausting the host. They are the OS-level defense against fork bombs, connection-exhaustion DoS, and file-descriptor leaks. Typical cases: Nginx high concurrency hitting Too many open files; a business process opening tens of thousands of connections and dragging down the box; a malicious or buggy script forking until the host looks dead; multi-user hosts that must cap processes per user. Correct limits contain the bad process instead of taking the whole server.

Prerequisites

• Linux server with root or sudo. This article uses CentOS 7+/Ubuntu 20+ (systemd)
• Know the process's sane resource waterline (start with `ulimit -a` and monitoring)
• For containers or cloud VMs, confirm the host does not override limits

How It Works

ulimit is Linux's soft/hard resource cap on processes. soft limit is the current value; a process may raise it up to the hard limit. hard limit is the ceiling; unprivileged processes may only lower it. Key items: nofile (open file descriptors; network connections consume this quota), nproc (process count), maxlogins (concurrent logins per account). Three config layers: shell sessions via `/etc/security/limits.conf` (PAM pam_limits); systemd services via unit LimitNOFILE and related directives; systemd user sessions via DefaultLimitNOFILE in `/etc/systemd/system.conf`. Layers do not inherit. Configuring only one is a common miss.

Procedure

Step 1: Inspect current limits

# All limits for the current shell
ulimit -a
# File descriptors only
ulimit -n
# Actual limits of a running process (e.g. Nginx)
cat /proc/$(pidof nginx | awk '{print $1}')/limits

Step 2: Set the limits.conf global baseline

Edit `/etc/security/limits.conf`, append (tune numbers to the workload):

# File descriptors: raise for service accounts to avoid high-concurrency errors
*    soft  nofile     65535
*    hard  nofile     65535
root soft  nofile     65535
root hard  nofile     65535
# Process count: cap ordinary users' fork ability; mitigates fork bombs
*    hard  nproc      4096
# Concurrent logins per account
*    hard  maxlogins  10

Confirm PAM loads the limits module. `/etc/pam.d/login` and `/etc/pam.d/common-session` (Ubuntu) should contain:

session required pam_limits.so

Step 3: Set systemd per-service limits

limits.conf does not apply to systemd-started services. Configure them separately. Nginx example, create an override:

systemctl edit nginx
# In the editor, write:
[Service]
LimitNOFILE=65535
LimitNPROC=4096
# Save, reload, restart
systemctl daemon-reload
systemctl restart nginx

Also set global defaults. Edit `/etc/systemd/system.conf`:

DefaultLimitNOFILE=65535
DefaultLimitNPROC=4096
# Requires a reboot to take effect

Step 4: Couple Nginx config for critical services

After raising OS limits, raise the app layer or Nginx still uses the old value:

# /etc/nginx/nginx.conf global section
worker_rlimit_nofile 65535;
events {
    worker_connections 10240;
}

Configuration Verification

# Re-login, then verify shell limits
ulimit -n
ulimit -nH
# Verify the systemd service's effective value
cat /proc/$(pidof nginx | awk '{print $1}')/limits | grep "open files"
# Load-test whether connections break the old cap (example: ab or in-app test)
ab -n 10000 -c 2000 http://127.0.0.1/
# Check for denials caused by the limit
journalctl -u nginx --since "10 min ago" | grep -i "too many"

FAQ

FAQ 1: After editing limits.conf, ulimit -n is still 1024?

Check in order: you must re-login (or restart the service); existing sessions do not hot-reload. systemd-started services read LimitNOFILE from the unit, not limits.conf. sshd sessions also need pam_limits.so in `/etc/pam.d/sshd`.

FAQ 2: nproc is too tight and the app fails to fork?

Confirm the account hitting the cap via `journalctl -k | grep -i fork` or app logs. Raise that account only. limits.conf supports per-user lines, e.g. appuser hard nproc 16384. Do not set global nproc to unlimited. That disables the fork-bomb defense.

Summary

ulimit is the lightest resource isolation. No extra software. First line of defense against resource exhaustion. Deploy: set nofile to concurrency needs and sync Nginx worker_rlimit_nofile; keep a global nproc ceiling against fork bombs; systemd services need their own LimitNOFILE. After config, verify each service via /proc/<pid>/limits. Put the check on the inspection list.