etcd Security Hardening in Practice: TLS Encryption and Access-Control Configuration Guide

Default etcd is world-readable. Compromise in a Kubernetes cluster leaks every secret. This guide hardens etcd with mutual TLS, RBAC, and network isolation.

etcd hardening is core to Kubernetes and distributed-system defense. etcd is the cluster KV hub: ServiceAccount secrets, ConfigMaps, Pod specs, all sensitive data. No auth, and one curl dumps the cluster. This guide covers mutual TLS, RBAC, and network isolation.

Use Cases

• Kubernetes etcd with no auth; unauthorized access risk
• Production etcd needs transport encryption and authentication
• Multi-tenant or multi-operator setups need permission isolation on etcd ops
• Security baseline (e.g. kube-bench) etcd checks failing

Prerequisites

• etcd 3.4+ (including kubeadm kube-system/etcd containers)
• OpenSSL 1.1+ for cert generation
• root on the node that holds the etcd data dir

How It Works

etcd security is three layers: TLS encrypts client↔etcd and peer traffic; `--client-cert-auth` forces a CA-issued client cert (mutual TLS); RBAC then gates read/write by user and role. Goal: no cert, no entry; cert without grants, no extra rights. kubeadm clusters enable peer certs by default; client side may still be one-way TLS. Complete it with this guide.

Procedure

1. Generate CA and certs

mkdir -p /etc/etcd/pki && cd /etc/etcd/pki
# 1) Create a local CA
openssl genrsa -out ca.key 2048
openssl req -x509 -new -nodes -key ca.key -days 3650 -subj "/CN=etcd-ca" -out ca.crt
# 2) Issue the server certificate (set IP to this host; SAN must include the domain/IP clients use to connect)
openssl genrsa -out server.key 2048
cat > server.cnf <<'EOF'
[req]
distinguished_name=dn
req_extensions=v3_req
[dn]
[alt_names]
IP.1 = 127.0.0.1
IP.2 = 10.0.0.10
EOF
openssl req -new -key server.key -subj "/CN=etcd-server" -out server.csr
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial   -days 3650 -out server.crt -extensions v3_req -extfile server.cnf
# 3) Issue the peer certificate (one per etcd node; CN is the node name)
openssl genrsa -out peer.key 2048
openssl req -new -key peer.key -subj "/CN=etcd-node1" -out peer.csr
openssl x509 -req -in peer.csr -CA ca.crt -CAkey ca.key -CAcreateserial   -days 3650 -out peer.crt
chmod 600 *.key

2. etcd startup flags (mutual TLS)

etcd \
  --name etcd-node1 \
  --data-dir /var/lib/etcd \
  --listen-client-urls https://10.0.0.10:2379,https://127.0.0.1:2379 \
  --advertise-client-urls https://10.0.0.10:2379 \
  --client-cert-auth --trusted-ca-file=/etc/etcd/pki/ca.crt \
  --cert-file=/etc/etcd/pki/server.crt --key-file=/etc/etcd/pki/server.key \
  --listen-peer-urls https://10.0.0.10:2380 \
  --peer-client-cert-auth --peer-trusted-ca-file=/etc/etcd/pki/ca.crt \
  --peer-cert-file=/etc/etcd/pki/peer.crt --peer-key-file=/etc/etcd/pki/peer.key

Critical flags: client-cert-auth for client cert checks, peer-client-cert-auth for peer cert checks. Both required.

3. Verify unauthorized access is denied

# Access without a certificate should fail
curl --cacert /etc/etcd/pki/ca.crt https://10.0.0.10:2379/version
# Expected: certificate error or handshake failure
# Access with a certificate succeeds
curl --cacert /etc/etcd/pki/ca.crt --cert /etc/etcd/pki/server.crt --key /etc/etcd/pki/server.key \
     https://10.0.0.10:2379/version

4. Enable RBAC users and roles

export ETCDCTL_API=3
alias ec="etcdctl --endpoints=https://10.0.0.10:2379 --cacert=/etc/etcd/pki/ca.crt --cert=/etc/etcd/pki/server.crt --key=/etc/etcd/pki/server.key"
# Create the root user and enable authentication
ec user add root
ec auth enable
# Create a read-only user and role
ec role add read-only
ec role grant-permission read-only read / --prefix=true
ec user add viewer
ec user grant-role viewer read-only
# Test: viewer can read but not write
ec --user viewer:password get / --prefix=true --limit=5
ec --user viewer:password put /test x
# Expected: write returns 13 (permission denied)

After auth enable, calls with no user return 401. Expected.

5. Firewall 2379 sources

# Allow only kube-apiserver nodes and the ops network segment to access 2379
firewall-cmd --permanent --add-rich-rule='rule family=ipv4 source address=10.0.0.20 port port=2379 protocol=tcp accept'
firewall-cmd --permanent --add-rich-rule='rule family=ipv4 port port=2379 protocol=tcp reject'
firewall-cmd --reload

6. Kubernetes: sync kube-apiserver

# kube-apiserver startup flags (Kubeadm: edit /etc/kubernetes/manifests/kube-apiserver.yaml)
--etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
--etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
--etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
--etcd-servers=https://10.0.0.10:2379

Verification

• curl 2379 with no cert fails cert check; with cert returns {etcdserver:3.5.x}
• `ec user list` shows root and viewer; `ec role list` shows read-only
• viewer get succeeds; put returns permission denied
• After firewall rules, non-allowlisted sources to 2379 are refused
• kubeadm cluster: `kubectl get pods -A` confirms apiserver↔etcd is healthy

FAQ

FAQ 1: Certs about to expire. How to rotate?

Re-issue longer-lived server/peer certs from the same CA, replace, restart etcd. No CA rebuild. If the CA itself expired, re-issue for every etcd node and every client (kube-apiserver, etcdctl). Major change. Roll in a maintenance window: peers first, clients second.

FAQ 2: etcdctl returns 401 after RBAC. Is data corrupt?

No. Auth is blocking. Re-auth as root: `etcdctl user get root`, or check etcd logs for the 401 source. Do not delete root or lose the password; recovery is `--auth-token` or snapshot restore.

FAQ 3: Cloud-managed etcd (ACK, EKS). Still harden?

Control-plane etcd is the vendor's job. On the customer side: confirm apiserver RBAC is on, do not grant `system:masters` casually, restrict write on kube-system, re-run kube-bench on etcd items.

Summary

etcd hardening is encrypt, authenticate, authorize, isolate: mutual TLS kills plaintext and forged clients; RBAC minimizes privilege; firewall shrinks 2379 to trusted nets. Production clusters must finish TLS and RBAC before exposing 2379. Put cert rotation on the ops calendar. That closes the cluster data-leak path at the root.