← News
/PLAYBOOK · 2026-07-27

ELK Stack Log Analysis Primer: Build a Security Event Monitoring Platform

Stand up an ELK Stack security log platform from zero. Install and configure Elasticsearch, Logstash, Kibana, and Filebeat. Centralize Nginx access logs and system auth logs, tag attacks, and visualize them.

Scope

ELK Stack (Elasticsearch, Logstash, Kibana) is the most widely used open-source log analysis platform. Fits SOC build-out, centralized server log management, attack tracing, and compliance audit. This guide is for security ops, pentesters, and sysadmins who need a security log analysis stack fast.

Prerequisites

Mechanism

ELK Stack has three parts: Elasticsearch (distributed search engine — store and query logs), Logstash (data pipeline — collect, parse, forward), and Kibana (viz platform — dashboards and alerts). Pair with Filebeat (lightweight shipper) for a full ELK + Filebeat architecture. In security analysis, ELK centralizes Nginx access logs, system auth logs (/var/log/auth.log), firewall logs, and more. Kibana dashboards give real-time attack visibility.

Procedure

Step 1: Install Elasticsearch

`# Import the Elasticsearch GPG key
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg

# Add the APT repository
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list

# Install
sudo apt update && sudo apt install -y elasticsearch

# Configure Elasticsearch (recommended settings for security log analysis)
sudo tee -a /etc/elasticsearch/elasticsearch.yml << 'EOF'
cluster.name: security-log-cluster
node.name: node-1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 127.0.0.1
http.port: 9200
xpack.security.enabled: false
discovery.type: single-node
EOF

# Start the service
sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch
# Verify
curl http://127.0.0.1:9200`

Step 2: Install Kibana

`sudo apt install -y kibana
sudo tee -a /etc/kibana/kibana.yml << 'EOF'
server.port: 5601
server.host: "127.0.0.1"
elasticsearch.hosts: ["http://127.0.0.1:9200"]
EOF

sudo systemctl enable kibana
sudo systemctl start kibana
# Verify: curl http://127.0.0.1:5601/api/status`

Step 3: Install Logstash

`sudo apt install -y logstash

# Create the Nginx log parsing config
sudo tee /etc/logstash/conf.d/nginx-security.conf << 'EOF'
input {
  beats {
    port => 5044
  }
}

filter {
  if [fileset][module] == "nginx" {
    grok {
      match => { "message" => "%{IPORHOST:client_ip} - %{DATA:user_name} [%{HTTPDATE:timestamp}] \"%{WORD:method} %{DATA:request} HTTP/%{NUMBER:http_version}\" %{NUMBER:response_code} %{NUMBER:bytes} \"%{DATA:referrer}\" \"%{DATA:user_agent}\"" }
    }
    date {
      match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
    }
    # Extract attack-signature fields
    geoip {
      source => "client_ip"
      target => "geo"
    }
    # Tag suspicious requests
    if [response_code] == 403 or [response_code] == 401 {
      mutate { add_tag => [ "blocked_request" ] }
    }
    if [request] =~ /(?:select|union|exec|insert|script|alert|eval)/i {
      mutate { add_tag => [ "attack_attempt" ] }
    }
  }
}

output {
  elasticsearch {
    hosts => ["127.0.0.1:9200"]
    index => "nginx-logs-%{+YYYY.MM.dd}"
  }
}
EOF

sudo systemctl enable logstash
sudo systemctl start logstash`

Step 4: Install Filebeat to Collect Nginx Logs

`sudo apt install -y filebeat

# Configure Filebeat
sudo tee /etc/filebeat/filebeat.yml << 'EOF'
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  fields:
    log_type: nginx_access
  # Multiline match: merge log lines from the same request
  multiline.pattern: '^\\d{1,3}\\.\\d{1,3}\\.'
  multiline.negate: true
  multiline.match: after

output.logstash:
  hosts: ["127.0.0.1:5044"]
EOF

sudo systemctl enable filebeat
sudo systemctl start filebeat`

Step 5: Collect System Auth Logs

`# Add auth.log collection config
sudo tee -a /etc/logstash/conf.d/auth-security.conf << 'EOF'
input {
  file {
    path => "/var/log/auth.log"
    type => "auth"
    start_position => "beginning"
  }
}

filter {
  if [type] == "auth" {
    grok {
      match => { "message" => "%{SYSLOG5424PRI:priority}?%{SYSLOGLINE}" }
    }
    # Tag SSH brute-force attempts
    if "Failed password" in [message] {
      mutate { add_tag => [ "ssh_bruteforce" ] }
    }
    if "Accepted password" in [message] {
      mutate { add_tag => [ "ssh_success" ] }
    }
  }
}

output {
  elasticsearch {
    hosts => ["127.0.0.1:9200"]
    index => "auth-logs-%{+YYYY.MM.dd}"
  }
}
EOF

sudo systemctl restart logstash`

Step 6: Create a Security Dashboard in Kibana

`# Create index patterns via API (for Kibana)
curl -X POST "http://127.0.0.1:5601/api/saved_objects/index-pattern" \
  -H "Content-Type: application/json" \
  -H "kbn-xsrf: true" \
  -d '{
    "attributes": {
      "title": "nginx-logs-*",
      "timeFieldName": "@timestamp"
    }
  }'

curl -X POST "http://127.0.0.1:5601/api/saved_objects/index-pattern" \
  -H "Content-Type: application/json" \
  -H "kbn-xsrf: true" \
  -d '{
    "attributes": {
      "title": "auth-logs-*",
      "timeFieldName": "@timestamp"
    }
  }'`

Verification

`# Verify Elasticsearch indices
curl -X GET "http://127.0.0.1:9200/_cat/indices?v"

# Verify that logs were written
curl -X GET "http://127.0.0.1:9200/nginx-logs-$(date +%Y.%m.%d)/_search?pretty" -d '{"size": 5}' -H "Content-Type: application/json"

# Count attack tags (401/403 requests)
curl -X GET "http://127.0.0.1:9200/nginx-logs-*/_count" -d '{
  "query": { "term": { "tags": "blocked_request" } }
}' -H "Content-Type: application/json"`

FAQ

Q1: Filebeat cannot connect to Logstash (Connection refused). What now?

Check three things: 1) Logstash running: `sudo systemctl status logstash`; 2) Logstash listen port: `sudo netstat -tlnp | grep 5044`; 3) Firewall: `sudo ufw status`. If Logstash is not bound to 0.0.0.0, add `host => "0.0.0.0"` in the input config.

Q2: High log volume fills Elasticsearch disk. How to handle?

Implement index lifecycle management (ILM): configure an ILM policy in Elasticsearch to roll indices to read-only after 7 days and delete after 30 days. Or a cron job: `0 3 * * * curl -X DELETE "http://127.0.0.1:9200/nginx-logs-$(date -d '30 days ago' +%Y.%m.%d)"`. Also sample high-traffic sites at the Filebeat layer (`sampling: 0.1`).

Takeaway

ELK Stack is solid infrastructure for security log analysis. Centralize Nginx access logs and system auth logs, then Grok parse, GeoIP locate, and auto-tag attacks. Security teams get real-time attack detection and source tracing. Next: Kibana Alert rules — email or DingTalk on SSH brute force (10+ failed logins in 5 minutes) and SQL injection attempts. Move from observation to active defense.