Prometheus Security Hardening: Unauthorized Access Protection Configuration Guide

Covers common Prometheus unauthorized-access risks and hardening: Basic Auth, TLS, reverse proxy, and management-API protection. Full configs and verification commands to stop metric leakage and RCE via reload/remote-write.

Scope

Prometheus hardening applies to every enterprise that exposes monitoring. Default Prometheus listens on 0.0.0.0:9090 with no authentication. Internet or intranet attackers can query business metrics, read target configs, and trigger remote rule-file loads (higher risk when web.enable-remote-write-receiver is enabled). This guide targets production environments where monitoring is externally reachable, MLPS requirements for management-interface authentication, and teams that want a unified entry for multiple Prometheus instances.

Prerequisites

• Prometheus 2.30+ deployed (commands verified on 2.45+)
• openssl and htpasswd on the Prometheus host (apache2-utils or httpd-tools)
• Reserved domain, or IP plus HTTPS port (default 9090)
• Nginx 1.18+ preinstalled if using the reverse-proxy option

Mechanism

Prometheus provides no user authentication. Official hardening is three layers: first, basic_auth so Prometheus enables HTTP Basic Auth on /metrics, /-/reload, and other endpoints; second, TLS via web.config.file, upgrading plaintext HTTP to HTTPS; third, reverse-proxy access control that also hides management endpoints. web.enable-lifecycle allows HTTP POST /-/reload to reload config. Unprotected, that hands config control to an attacker. Pair it with authentication.

Procedure

Step 1: Generate password hash and TLS certificate

# 1) Generate Basic Auth password hash (enter password twice; record output)
htpasswd -nBC 12 "" | tr -d ':\n'

# 2) Generate self-signed certificate (use a proper cert in production)
mkdir -p /etc/prometheus/certs
openssl req -newkey rsa:2048 -nodes \
  -keyout /etc/prometheus/certs/prometheus.key \
  -out /etc/prometheus/certs/prometheus.csr \
  -subj "/CN=prometheus.example.com"
openssl x509 -req -in /etc/prometheus/certs/prometheus.csr \
  -signkey /etc/prometheus/certs/prometheus.key \
  -out /etc/prometheus/certs/prometheus.crt -days 365

Step 2: Create web config file

Write the hash from the previous step into /etc/prometheus/web.yml:

# /etc/prometheus/web.yml
basic_auth_users:
  admin: $2y$12$K0iH... (paste bcrypt hash from step 1)
tls_server_config:
  cert_file: /etc/prometheus/certs/prometheus.crt
  key_file: /etc/prometheus/certs/prometheus.key
  client_auth_type: "NoClientCert"

Step 3: Enable auth and TLS via startup flags

# Edit ExecStart in systemd unit /etc/systemd/system/prometheus.service
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --web.config.file=/etc/prometheus/web.yml \
  --web.listen-address=0.0.0.0:9090

systemctl daemon-reload && systemctl restart prometheus

Step 4: Reverse proxy (optional; unified entry for multiple instances)

# /etc/nginx/conf.d/prometheus.conf
server {
    listen 443 ssl;
    server_name prometheus.example.com;
    ssl_certificate     /etc/prometheus/certs/prometheus.crt;
    ssl_certificate_key /etc/prometheus/certs/prometheus.key;

    location / {
        auth_basic "Prometheus";
        auth_basic_user_file /etc/nginx/.htpasswd;
        # Allow intranet CIDR only
        allow 10.0.0.0/8;
        deny all;
        proxy_pass http://127.0.0.1:9090;
        proxy_set_header Authorization $http_authorization;
    }
}
nginx -t && systemctl reload nginx

Verification

# 1) Unauthenticated request should return 401
curl -i http://127.0.0.1:9090/metrics | head -1
# Expected: HTTP/1.1 401 Unauthorized

# 2) Authenticated request should succeed
curl -u admin:password https://prometheus.example.com/metrics | head -5

# 3) Verify TLS certificate is in effect
openssl s_client -connect 127.0.0.1:9090 -servername prometheus.example.com 2>/dev/null | grep -i subject

FAQ

FAQ 1: After enabling basic_auth, can scrape targets (node_exporter) still work?

Prometheus basic_auth only protects the Web UI and management APIs. It does not affect Prometheus pulling exporter metrics. If the exporter itself has auth enabled, configure it in prometheus.yml scrape_config:

scrape_configs:
- job_name: node
  static_configs: [{ targets: ["10.0.0.5:9100"] }]
  basic_auth:
    username: exporter_user
    password: exporter_pass

FAQ 2: Should web.enable-remote-write-receiver and web.enable-lifecycle be enabled?

Both enlarge the attack surface. Keep them off by default. Enable web.enable-lifecycle only when HTTP hot-reload is required, and only after basic_auth is confirmed working. Enable web.enable-remote-write-receiver only when accepting remote write (Prometheus federation or agent mode), and restrict via firewall to trusted sources. If 401 verification fails after enablement, check that the bcrypt hash in web.yml starts with `$2y$` (htpasswd -B outputs `$2y$`).

Summary

Unauthorized Prometheus access is the most common real-world risk in monitoring stacks. Basic Auth + TLS + optional reverse proxy turns the default open 9090 port into an authenticated HTTPS service. Three landing notes: password hash must use -B (bcrypt); verify 401 first, then 200; enable remote-write and lifecycle APIs only as needed. That keeps Prometheus exposure within an acceptable range.