/PLAYBOOK · 2026-09-15

Memory Forensics in Practice: Tracing Intrusions with Volatility

Memory forensics finds hidden processes, fileless malware, and outbound connections that disk forensics miss. This article covers LiME and AVML acquisition commands, SHA256 evidence sealing, Volatility 3 pslist/malfind/sockstat/bash analysis, and production-impact controls.

Memory forensics extracts process, network-connection, and kernel-module evidence directly from physical RAM during incident response. It finds hidden processes, fileless malware, and wiped traces that disk forensics miss. This article gives complete Linux memory-acquisition (LiME / AVML), hash-sealing, Volatility 3 analysis, and artifact-extraction commands, and how to freeze evidence inside a business-acceptable window.

Scope

Server looks compromised, but logs are wiped and no webshell or malicious binary is on disk.
Anomalous outbound connections or CPU usage vanish after reboot and cannot be reproduced on disk.
Hunting fileless attacks (in-memory webshells, reflective loaders, `/proc` injection) and kernel rootkits.
Need verifiable electronic evidence for attribution, loss assessment, or regulatory reporting.

Prerequisites

Root on the server. Confirm the host is not so thoroughly owned that forensic tools themselves can be tampered with.
`kernel-devel` / `linux-headers` matching the running kernel to build LiME. Use the AVML static binary if you cannot compile.
Writable external storage (USB HDD or a separate mount). Size ≥ 2× physical RAM.
Analysis box with Volatility 3 and Python 3.8+.
Assess business impact first: acquisition briefly stalls some syscalls. Run off-peak and notify the business owner.

Mechanism

RAM is volatile evidence and sits at the top of the collection order. RFC 3227: registers and cache, memory, tmpfs, disk, remote logs, physical media, archives. Power-off or reboot permanently destroys process lists, network connections, decryption keys, and command history.

Three advantages over disk forensics:

Hidden objects stay visible. Syscall hooks or kernel-structure edits can fool on-disk `ps`; the in-memory process list and kernel objects still hold the original traces. Cross-compare to find mismatches.
Capture fileless payloads. In-memory webshells, reflectively loaded ELFs, and code executed via `memfd_create` never hit disk. Extract them from RAM only.
Recover keys and plaintext credentials. Volume or config master keys often reside in memory and can be recovered from the dump.

Three common Linux acquisition methods:

LiME: loadable kernel module that reads physical memory and writes `.lime`. Low intrusion. Preferred.
AVML: Microsoft static binary, no kernel module. Writes `.raw`.
Direct copy of `/proc/kcore`: emergency-only. ELF header and access limits require extra conversion. Weaker integrity.

Procedure

Step 1: Isolate the environment and mount the evidence disk

# Record time as the forensic timeline start
date -u +"%Y-%m-%dT%H:%M:%SZ"

# Mount external evidence disk (example /dev/sdb1)
mkdir -p /mnt/evidence
mount -o ro /dev/sdb1 /mnt/evidence   # First capture needs write access: drop ro, but do not write to the system disk

# Minimize FS writes so other evidence stays intact
echo 3 > /proc/sys/vm/drop_caches

Step 2: Pre-acquisition snapshot (volatile data first)

OUT=/mnt/evidence/$(hostname)-$(date +%Y%m%d-%H%M%S)
mkdir -p "$OUT"

# Network connections and listening ports (with PID)
ss -antp > "$OUT/ss_antp.txt" 2>&1
netstat -antp >> "$OUT/netstat.txt" 2>&1
lsof -i -n -P > "$OUT/lsof_net.txt" 2>&1

# Processes and kernel modules
ps auxwwf > "$OUT/ps_auxwwf.txt"
lsmod > "$OUT/lsmod.txt"
cat /proc/modules > "$OUT/proc_modules.txt"

# Logins and command history
last -Fa > "$OUT/last.txt"; lastb -Fa > "$OUT/lastb.txt" 2>&1
w; who -a > "$OUT/who.txt"
for d in /home/* /root; do
  [ -f "$d/.bash_history" ] && cp "$d/.bash_history" "$OUT/$(basename $d)_bash_history.txt"
done

This does not replace a memory image. It is cheap and later validates memory-analysis conclusions.

Step 3: Capture the memory image (LiME)

# 1. Install headers matching the running kernel
yum install -y kernel-devel-$(uname -r) gcc make
# Debian / Ubuntu
apt-get install -y linux-headers-$(uname -r) build-essential

# 2. Fetch and compile LiME
git clone https://github.com/504ensicsLabs/LiME.git /opt/LiME
cd /opt/LiME/src && make

# 3. Capture memory, write to the evidence disk
insmod /opt/LiME/src/lime-$(uname -r).ko \
  "path=$OUT/memory.lime format=lime"
# Capture time depends on RAM size and disk write speed; unload the module when finished
rmmod lime

ls -lh "$OUT/memory.lime"

Key parameters: `format=lime` is the recommended format; compatible with Volatility and converters. `path` must point at the evidence disk so you do not fill the system volume and interrupt production. `digest=sha256` can hash during capture on some versions.

If you cannot build the kernel module, use AVML:

curl -L -o /opt/avml \
  https://github.com/microsoft/avml/releases/latest/download/avml
chmod +x /opt/avml
/opt/avml "$OUT/memory.raw"
ls -lh "$OUT/memory.raw"

Step 4: Seal evidence integrity

cd "$OUT"
sha256sum memory.lime > memory.lime.sha256 2>/dev/null || \
sha256sum memory.raw > memory.raw.sha256
sha256sum *.txt >> volatile-files.sha256

# Record OS and hardware metadata for later profile matching
uname -a > system-info.txt
cat /etc/os-release >> system-info.txt
dmidecode -t system >> system-info.txt 2>/dev/null
lscpu >> system-info.txt

# Pack and re-verify
cd /mnt/evidence && tar czf $(basename $OUT).tar.gz $(basename $OUT)
sha256sum $(basename $OUT).tar.gz > $(basename $OUT).tar.gz.sha256
ls -lh /mnt/evidence

Step 5: Volatility 3 analysis

pip3 install volatility3
vol3 -h | head -5

# Auto-identify the memory image (Volatility 3 usually needs no manual profile)
vol3 -f memory.lime banners.Banners

# Process tree: focus on unusual parents and pathless processes
vol3 -f memory.lime linux.pslist.PsList | tee pslist.txt
vol3 -f memory.lime linux.pstree.PsTree | tee pstree.txt

# Detect injection traces in process memory maps
vol3 -f memory.lime linux.malfind.Malfind | tee malfind.txt

# Kernel modules and syscall table (hunt for rootkits)
vol3 -f memory.lime linux.lsmod.Lsmod | tee lsmod_vol.txt
vol3 -f memory.lime linux.check_syscall.Check_syscall

# Network connections (no need for on-disk netstat)
vol3 -f memory.lime linux.sockstat.Sockstat | tee sockstat.txt

# Command history and bash sessions
vol3 -f memory.lime linux.bash.Bash | tee bash_history.txt

Compare `linux.pslist.PsList` line-by-line with the pre-capture `ps_auxwwf.txt`. Processes present in memory but missing from `ps`, or listed by `ps` but missing from memory, need independent tracing. Dump `linux.malfind.Malfind` hits and YARA-scan them:

vol3 -f memory.lime -o dump linux.malfind.Malfind --dump
# Match malware signatures against dumped memory segments
yara -r /opt/rules/malware_index.yar dump/ | tee yara_hits.txt

Step 6: Extract persistence and credential clues

# Search memory for key and credential clues (run against exported strings)
vol3 -f memory.lime linux.pagecache.InodePages --dump 2>/dev/null
strings -a memory.lime | grep -aiE "BEGIN (RSA|OPENSSH|EC) PRIVATE KEY" | head
strings -a memory.lime | grep -aiE "password|secret|token" | sort -u | head -50

Cross-check hits against the asset inventory so you do not treat the application's own config as stolen credentials.

Verification

Image integrity: `sha256sum -c memory.lime.sha256` prints `OK`. Image size should be 80%–100% of physical RAM (compression and page cache affect this).
Acquisition validity: `vol3 -f memory.lime banners.Banners` prints kernel version and distro. If unrecognized, the image format is wrong or the kernel is too new; try `--single-location` to point at symbols.
Analysis consistency: `linux.pslist.PsList` row count should be close to `ps -e` at capture time. Listening ports in `linux.sockstat.Sockstat` should match the `ss -antp` snapshot. Deltas are the anomalies you must explain.
Chain of custody: evidence directory contains the image, hashes, volatile snapshots, system metadata, and operator notes. Names include hostname and UTC timestamp.

FAQ

FAQ1: Will production memory capture interrupt the business?

LiME briefly freezes some pages for a consistent snapshot. Expect seconds to tens of seconds of stall, depending on RAM size and disk write speed. For high-concurrency online services, three mitigations: run off-peak; write to a dedicated fast disk or a temporarily mounted SSD so you do not contend with the data volume; if virtualized, snapshot VM memory on the hypervisor and skip loading a kernel module in the guest. If stall is unacceptable, fall back to `ss`, `lsof`, `/proc` snapshots plus `gcore` of key processes, then reboot into disk forensics later.

FAQ2: Capture failed or the image is far too small. How to debug?

LiME build fails: missing `kernel-devel` / `linux-headers` that exactly match the running kernel. Confirm with `uname -r` and reinstall. If the distro updated the kernel without a reboot, install the old headers or reboot first, then capture.
Output much smaller than RAM: disk full, abort. Confirm `dmesg | grep -i lime` has no errors. Some hypervisors omit unallocated pages; that is normal.
Volatility cannot identify the image: confirm `format=lime` (Volatility 3 handles `padded` poorly). Convert with `lime2raw` or `avml-convert` if needed.

FAQ3: How do I corroborate memory findings with logs and disk?

Three-source timeline: use in-memory process start times and connection establishment as anchors; find matching records in `journalctl` / `auth.log` / Nginx access logs; then verify the binary path and file hash on disk. Process in memory, file deleted on disk = drop-and-delete; extract the page-cache image with `linux.pagecache`. File on disk, no process in memory = likely a scheduled backdoor; check `crontab`, systemd timers, and `rc.local`. Close all three sources before writing the IR report and attribution.

Summary

Memory forensics is about the volatility window: after reboot, hidden processes, fileless payloads, and outbound records are often gone. Standard order: isolate → volatile snapshot → memory capture → hash seal → Volatility analysis → three-source corroboration. Hash sealing and system metadata are prerequisites for usable evidence. For production fleets, bake LiME, AVML, and Volatility 3 into image templates, predefine the evidence-disk mount and owner, and cut IR prep time to the minimum.