/PLAYBOOK · 2026-09-14

Log Audit in Practice: Promtail and Loki Deployment and Configuration

Log audit needs centralized retention and fast search. Deploy Promtail + Loki + Grafana with docker compose: three complete configs, label cardinality constraints, attack-signature LogQL (SQLi/scanners/SSH brute force), ruler alerts, 30-day retention, and end-to-end self-test.

Scope

Log audit is the base capability for intrusion detection and post-incident tracing. Many small and mid-size sites still grep single-host logs. When something breaks, you SSH around machines and dig through rotated files; often the trail is gone before you locate the attacker. This article ships a log-audit stack that runs on one host or a small cluster: Promtail collects, Loki stores and indexes, Grafana queries and alerts. All components run as containers.

This stack indexes labels only, not log bodies. Storage and memory cost is far below full-text ELK. It fits daily log growth in the GB range with long retention (30+ days). Typical uses: attack-signature search in Nginx access logs, SSH and auth-failure analysis, WebShell landing traces, full timelines by IP or URL, anomaly-rate auto-alerts.

If you need complex full-text search and large-scale aggregation, use ELK. For centralized retention + fast label lookup + lightweight alerting, Loki is cheaper to deploy and operate.

Prerequisites

One Linux server with Docker 20.10+ and Docker Compose v2.
At least 20 GB free disk (size from retention volume and compression ratio).
Log paths and formats known (this article uses Nginx access.log and auth.log).
Clock synced (chrony or systemd-timesyncd). Skewed timestamps break timeline reconstruction.
Expose Grafana only. Loki and Promtail stay on the internal network.

Internals

Split the three roles first:

Promtail is the collector. It discovers files, applies labels, pushes streams. It replaces generic shippers and speaks Loki's push protocol natively.
Loki is storage and query. It writes streams keyed by label set + timestamp to object storage or local disk, and builds an inverted index on labels only.
Grafana is query UI and alerting. LogQL searches logs; metric queries drive alerts.

The hard constraint is label cardinality. Labels enter the index. High-cardinality fields such as `ip`, `request_uri`, `user_agent` must never be labels, or the index balloons and can take Loki down. Keep only low-cardinality labels: `job`, `host`, `service`, `log_type`. Extract high-cardinality fields at query time with parsers (`| pattern`, `| json`).

That is Loki's query model: match streams by labels first (at least one label matcher is required), then filter and parse line by line. Query cost scales with hit volume, so label design sets query speed.

Procedure

Step 1: Write the docker-compose file

mkdir -p /opt/log-audit && cd /opt/log-audit
mkdir -p loki-data grafana-data promtail-positions loki-rules

# /opt/log-audit/docker-compose.yml
services:
  loki:
    image: grafana/loki:3.1.0
    container_name: loki
    restart: unless-stopped
    command: -config.file=/etc/loki/loki.yml
    volumes:
      - ./loki.yml:/etc/loki/loki.yml:ro
      - ./loki-data:/loki
      - ./loki-rules:/etc/loki/rules:ro
    ports:
      - "127.0.0.1:3100:3100"

  promtail:
    image: grafana/promtail:3.1.0
    container_name: promtail
    restart: unless-stopped
    command: -config.file=/etc/promtail/promtail.yml
    volumes:
      - ./promtail.yml:/etc/promtail/promtail.yml:ro
      - ./promtail-positions:/var/positions
      - /var/log:/var/log:ro          # read-only mount; collector must not mutate source logs
    depends_on:
      - loki

  grafana:
    image: grafana/grafana:11.1.0
    container_name: grafana
    restart: unless-stopped
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=ChangeMe_Strong_Pwd
      - GF_USERS_ALLOW_SIGN_UP=false
    volumes:
      - ./grafana-data:/var/lib/grafana
    ports:
      - "3000:3000"
    depends_on:
      - loki

Notes: bind Loki to 127.0.0.1, not the public interface. Mount `/var/log` ro so the collector cannot rewrite logs. Persist `promtail-positions` or a restart replays entire files.

Step 2: Write the Loki config

# /opt/log-audit/loki.yml
auth_enabled: false
server:
  http_listen_port: 3100
  grpc_listen_port: 9095
common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory
schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h
limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h        # reject samples older than 7 days; suppress out-of-order writes
  max_query_series: 5000
  retention_period: 720h                  # retain logs 30 days
compactor:
  working_directory: /loki/compactor
  retention_enabled: true                 # retention is enforced only by the compactor
  delete_request_store: filesystem
ruler:
  storage:
    type: local
    local:
      directory: /etc/loki/rules
  rule_path: /loki/rules-temp
  enable_api: true

Three knobs matter: `retention_period: 720h` is the 30-day audit hold. `retention_enabled: true` must be on or the policy is a no-op. `reject_old_samples_max_age` blocks stale samples when collector clocks drift.

Step 3: Write the Promtail scrape config

# /opt/log-audit/promtail.yml
server:
  http_listen_port: 9080
positions:
  filename: /var/positions/positions.yml
clients:
  - url: http://loki:3100/loki/api/v1/push
scrape_configs:
  # Nginx access log
  - job_name: nginx-access
    static_configs:
      - targets: [localhost]
        labels:
          job: nginx-access
          host: web-01
          log_type: access
          __path__: /var/log/nginx/access.log

  # Nginx error log, with multiline stack merge
  - job_name: nginx-error
    static_configs:
      - targets: [localhost]
        labels:
          job: nginx-error
          host: web-01
          log_type: error
          __path__: /var/log/nginx/error.log
    pipeline_stages:
      - multiline:
          firstline: '^\d{4}/\d{2}/\d{2} '
          max_wait_time: 3s

  # System auth log
  - job_name: syslog-auth
    static_configs:
      - targets: [localhost]
        labels:
          job: syslog-auth
          host: web-01
          log_type: auth
          __path__: /var/log/auth.log

Labels stay at three low-cardinality fields: job/host/log_type. That is the cardinality rule above. Parse IP at query time. Do not put it in `labels`.

Pin Nginx to a stable format so query-time parsers hold:

log_format audit '$remote_addr - $remote_user [$time_iso8601] '
                 '"$request" $status $body_bytes_sent "$http_referer" '
                 '"$http_user_agent" rt=$request_time uct=$upstream_connect_time';
access_log /var/log/nginx/access.log audit;

Step 4: Start and confirm ingest

cd /opt/log-audit
docker compose up -d
docker compose ps
docker compose logs --tail=30 promtail
docker compose logs --tail=30 loki

Verify streams on Loki directly:

# list known labels and values
curl -s http://127.0.0.1:3100/loki/api/v1/labels | head -c 400
curl -s 'http://127.0.0.1:3100/loki/api/v1/label/job/values'
curl -s 'http://127.0.0.1:3100/loki/api/v1/label/log_type/values'

Treat Promtail errors other than `level=info msg="entry added"` and `msg="compacting"` as incidents (429, 400). 429 is usually Loki ingest rate-limit. 400 is usually labels or timestamps.

Step 5: Wire Grafana and run audit queries

# add datasource (or add it in the Grafana UI)
curl -s -X POST http://admin:ChangeMe_Strong_Pwd@127.0.0.1:3000/api/datasources \
  -H 'Content-Type: application/json' \
  -d '{"name":"Loki","type":"loki","url":"http://loki:3100","access":"proxy","isDefault":true}'

Reusable LogQL for the common audit cases:

# 1) 5xx in the last 1h, clustered by URL
sum by (uri) (
  count_over_time({job="nginx-access"}
    | pattern `<_> - <_> [<_>] "<method> <uri> <_>" <status> <_>` 
    | status =~ "5.." [1h])
)

# 2) top 10 request IPs (aggregate after parse; do not label)
topk(10, sum by (ip) (
  count_over_time({job="nginx-access"}
    | pattern `<ip> - <_> [<_>] "<_>" <_> <_>` [1h])
))

# 3) suspected SQLi and path traversal
{job="nginx-access"} |~ `(?i)(union\\s+select|\\.\\./|%2e%2e|/etc/passwd|information_schema)`

# 4) scanner and common automation fingerprints
{job="nginx-access"} |~ `(?i)(nikto|sqlmap|nmap|masscan|dirbuster|acunetix|nessus|zgrab)`

# 5) SSH auth-failure spike, for brute-force alerts
sum(count_over_time({job="syslog-auth"} |= "Failed password" [5m]))

# 6) full timeline for one IP (tracing)
{job="nginx-access"} |= "203.0.113.45"

# 7) ingest rate, collector health
sum(rate({job="nginx-access"}[5m]))

Items 3 and 4 are the attack-signature core: regex scans inside the stream, no pre-built index, so new rules take effect immediately. Item 6 stitches one IP's actions by time; that is the usual tracing query.

Step 6: Configure auto-alerts

Loki ruler fires on LogQL metric queries. Drop rules in `loki-rules/`:

# /opt/log-audit/loki-rules/audit-alerts.yml
groups:
  - name: security-audit
    rules:
      - alert: SSHBruteForceSuspected
        expr: |
          sum by (host) (count_over_time({job="syslog-auth"} |= "Failed password" [5m])) > 50
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.host }} SSH auth failures exceeded 50 in 5 minutes"

      - alert: WebScanDetected
        expr: |
          sum(count_over_time({job="nginx-access"}
            |~ `(?i)(sqlmap|nikto|acunetix|nessus)` [5m])) > 10
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Scanner fingerprint requests detected; likely automated scan"

Reload after edits: `curl -X POST http://127.0.0.1:3100/loki/api/v1/rules/security-audit/reload`. Confirm load: `curl -s http://127.0.0.1:3100/loki/api/v1/rules`. In Grafana Alerting, point at the Loki datasource for richer notify paths (mail, webhook to internal IM).

Verification

# 1. all three containers running
docker compose ps --format 'table {{.Name}}\t{{.Status}}'

# 2. Loki ready
curl -s http://127.0.0.1:3100/ready

# 3. stream labels present
curl -s 'http://127.0.0.1:3100/loki/api/v1/label/job/values'

# 4. query returns last 5 records
curl -s -G http://127.0.0.1:3100/loki/api/v1/query_range \
  --data-urlencode 'query={job="nginx-access"}' \
  --data-urlencode 'limit=5' | head -c 600

# 5. emit a test log and confirm search (end-to-end path)
logger -p auth.info 'LOG_AUDIT_SELFTEST failed password for testuser'
curl -s -G http://127.0.0.1:3100/loki/api/v1/query_range \
  --data-urlencode 'query={job="syslog-auth"} |= "LOG_AUDIT_SELFTEST"'

# 6. Grafana alive and datasource reachable
curl -s http://127.0.0.1:3000/api/health

Order encodes the debug rule: confirm collector push, then Loki write, then the query. Step 5 is the high-value self-test: file perms, scrape rules, push path, and query parse. If step 5 misses and step 3 is fine, the usual cause is `__path__` or a mount mismatch inside the container.

FAQ

Q1: Loki memory keeps climbing as volume grows. How do you cap it?

Decide ingest pressure vs query pressure first. Ingest: check high-cardinality labels (`curl -s 'http://127.0.0.1:3100/loki/api/v1/label/<name>/values' | wc -c` as a rough value-count). Pull `ip`, `uri`, `user_agent` out of labels; parse at query time. Query: never run unbounded time ranges; cap with `limits_config.max_query_series` and `max_query_parallelism`; default Grafana panels to 1 hour, not 7 days. Slim Nginx logs too: drop unused `$http_referer`, `access_log off` on health-check paths. That often cuts 30%+ volume.

Q2: After a container restart, logs duplicate or a history gap appears. What then?

Duplicates mean the positions file was not persisted. Promtail records per-file offsets there. If `/var/positions` is not bind-mounted, a recreate resets offsets and replays from byte 0. Confirm `./promtail-positions:/var/positions` in compose and that the dir is writable. Gaps are usually rotation: Promtail handles rename+create, but nonstandard rotation (delete-then-create, inode reuse) needs pipeline `tail` tweaks, or dated files (`access-YYYY-MM-DD.log`) with a glob path.

Q3: Can this replace a WAF or IDS?

No. Different job. Loki is post-facto audit and visibility: evidence in one place, searchable, alertable. The request already hit the backend; success still needs a human. WAF and IDS intercept in-line, before the app. Correct stack is three layers: WAF at the edge for known signatures and high-rate attacks; host log audit for low-and-slow bypass; alerts to a human. Audit logs also feed WAF tuning — extract real bypassed request features from logs, add rules, close the loop.

Summary

Promtail + Loki + Grafana is a three-container log-audit path that stays up: scattered host logs become searchable, alertable, long-retained evidence. Four landing rules: low-cardinality labels only, high-cardinality at query time; persist positions and data dirs so restarts neither lose offsets nor replay; retention only works if the compactor is actually enabled; start alert thresholds conservative, watch a week of real cardinality, then tighten. After deploy, pair a unified Nginx log format with WAF rule feedback, and you stop grepping after the fact.