Kahibaro
Discord Login Register ☰
☰
Previous
Up
Next

Collecting evidence

Table of Contents

Incident response mindset for evidence collection

In forensics and incident response, evidence collection must balance speed, accuracy, and preservation. In practice this means:

This chapter focuses on what to collect, in which order, and how, with Linux-specific tools and practices.

Key principles:

Preparation and environment

Response workstation and toolset

Never rely solely on the compromised system. Ideally, use:

Before the incident:

During the incident:

Order of volatility and collection strategy

General order of volatility (most to least volatile):

  1. CPU state, registers, running processes, and memory (RAM).
  2. Network connections and ephemeral data (ARP, routing cache, netstat state).
  3. Logged-in users and sessions.
  4. Temporary files, /tmp, /run, swap.
  5. Application logs and system logs.
  6. Filesystem metadata (timestamps, extended attributes).
  7. Full disk images.
  8. Off-host logs and cloud metadata (less volatile but critical).

For Linux, a practical workflow:

  1. Stabilize and isolate (network isolation) while minimizing changes.
  2. Immediately start documentation (photos, notes, typed commands history).
  3. Collect live data:
    • System clock, uptime.
    • Logged-in users.
    • Running processes, open files.
    • Network state.
    • Memory image (if in scope and tools available).
  4. Collect key logs and configuration.
  5. Acquire disk-level images (or at least key partition images).
  6. Collect external logs (IDS, firewall, VPN, cloud provider, auth systems).

Documentation and chain of custody

Field notes

Document in real time:

Example: keep a text log on a separate device: 

# All timestamps in UTC
2025-04-01T13:20:10Z - IR lead: Alice
- Host: web01.example.com (10.0.5.12)
- Access over out-of-band console
- Command: /mnt/ir-tools/bin/ps -eo pid,ppid,cmd > /mnt/evidence/web01_ps.txt
- SHA256(web01_ps.txt)=...

If possible, capture:

Example using script (from a trusted location): 

/mnt/ir-tools/bin/script -t 2> /mnt/evidence/typescript.timing \
  /mnt/evidence/typescript.log

Chain of custody records

For each evidence item:

Hash example: 

sha256sum memdump.lime > memdump.lime.sha256
sha256sum disk-image.dd > disk-image.dd.sha256

Use consistent naming and immutable storage (e.g. WORM storage, object storage with versioning and retention policies) where possible.

Live data collection on Linux

Assume the system is still powered on. The challenge is to collect as much as possible without altering evidence more than necessary.

Time and system identity

First, capture reference information: 

# Use trusted binaries if available
date -u
hostname -f
uname -a
id
who
last -n 20

Also capture:

Running processes and system state

Collect comprehensive process and system info: 

ps auxww > /mnt/evidence/ps_aux.txt
ps -eo pid,ppid,user,group,start,time,command > /mnt/evidence/ps_detailed.txt
top -b -n 1 > /mnt/evidence/top_snapshot.txt
lsof -nP > /mnt/evidence/lsof_all.txt
lsmod > /mnt/evidence/lsmod.txt

Prefer -ww to avoid truncated command lines.

If procps tools are distrusted, read directly from /proc: 

cp -a /proc /mnt/evidence/proc_snapshot

(Be aware this is large and may change as you copy; note that in your documentation.)

Network connections and configuration

Network state is highly volatile; gather early: 

ip addr show > /mnt/evidence/ip_addr.txt
ip route show > /mnt/evidence/ip_route.txt
ip neigh show > /mnt/evidence/ip_neigh.txt
ss -tulpn > /mnt/evidence/ss_listen.txt
ss -tanp > /mnt/evidence/ss_tcp_all.txt
ss -uanp > /mnt/evidence/ss_udp_all.txt

If ss is unavailable: 

netstat -plant > /mnt/evidence/netstat_plant.txt
netstat -rn > /mnt/evidence/netstat_rn.txt

Capture firewall rules: 

iptables-save > /mnt/evidence/iptables_save.txt 2>/dev/null || true
ip6tables-save > /mnt/evidence/ip6tables_save.txt 2>/dev/null || true
nft list ruleset > /mnt/evidence/nft_ruleset.txt 2>/dev/null || true

Also note:

Users, sessions, and authentication state

Capture who is (and was recently) logged in: 

who > /mnt/evidence/who.txt
w > /mnt/evidence/w.txt
last -n 100 > /mnt/evidence/last.txt
id > /mnt/evidence/id_current_user.txt

Optionally gather:

Example: 

cp /etc/passwd /mnt/evidence/etc_passwd
cp /etc/group /mnt/evidence/etc_group
cp /etc/shadow /mnt/evidence/etc_shadow_restricted
chmod 600 /mnt/evidence/etc_shadow_restricted

Kernel and system configuration

Capture runtime kernel parameters: 

sysctl -a > /mnt/evidence/sysctl_all.txt 2>/dev/null || true
cat /proc/cmdline > /mnt/evidence/proc_cmdline.txt
dmesg > /mnt/evidence/dmesg.txt 2>/dev/null || true

Note SELinux/AppArmor state: 

getenforce > /mnt/evidence/selinux_enforce.txt 2>/dev/null || true
sestatus > /mnt/evidence/selinux_status.txt 2>/dev/null || true
aa-status > /mnt/evidence/apparmor_status.txt 2>/dev/null || true

Filesystem overview and suspicious artifacts

Collect a high-level snapshot of the filesystem without recursively dumping everything: 

df -hT > /mnt/evidence/df_hT.txt
mount > /mnt/evidence/mount.txt
lsblk -f > /mnt/evidence/lsblk_f.txt

Capture directory listings that help later triage: 

# Common executable and temp locations
ls -alR /bin /sbin /usr/bin /usr/sbin > /mnt/evidence/ls_system_bins.txt
ls -alR /tmp /var/tmp /dev/shm > /mnt/evidence/ls_temp_areas.txt
# Home directories
ls -alR /home > /mnt/evidence/ls_home.txt

Depending on size, you may restrict recursion, or use find to capture metadata only: 

find / -xdev -printf '%p|%u|%g|%m|%s|%TY-%Tm-%Td %TH:%TM:%TS\n' \
  > /mnt/evidence/find_metadata_root.txt 2>/dev/null

Memory acquisition on Linux

Memory acquisition is technically complex and kernel-version-dependent, but extremely valuable for:

Considerations before dumping RAM

Common acquisition approaches

  1. Kernel modules like LiME (Linux Memory Extractor).
  2. Hypervisor-level dumps for virtual machines (preferred when available).
  3. Hardware-based methods (outside scope here).

Using LiME (conceptual overview)

Typical steps (from a trusted source): 

  insmod lime.ko "path=/mnt/evidence/memdump.lime format=lime"
  rmmod lime

Then hash the image: 

sha256sum /mnt/evidence/memdump.lime > /mnt/evidence/memdump.lime.sha256

For VMs, use hypervisor tools (e.g. virsh dump for KVM, snapshot features in cloud providers) to get a memory snapshot without touching the guest, when possible. That approach is usually preferable for IR.

Disk and filesystem evidence

Triaging vs full imaging

You often will not start with a full disk image if rapid triage is needed, but you should understand when imaging is required:

When possible, aim for both: quick triage evidence now, and full imaging soon after.

Creating a disk/partition image with `dd` or `dcfldd`

Use a read-only source:

Example: 

# Identify target
lsblk -f
# Imaging with dd (careful with if/of order)
dd if=/dev/sda of=/mnt/evidence/disk-sda.img bs=4M conv=noerror,sync status=progress
# Hash the image
sha256sum /mnt/evidence/disk-sda.img > /mnt/evidence/disk-sda.img.sha256

dcfldd adds built-in hashing and split output: 

dcfldd if=/dev/sda hash=sha256 hashlog=/mnt/evidence/sda_hash.log \
  split=2000M of=/mnt/evidence/disk-sda.img.

Imaging virtual disks

For VMs, often you can:

Example (KVM/libvirt): 

# Stop or snapshot the VM (as policy allows)
virsh suspend vmname
# Copy the disk
cp /var/lib/libvirt/images/vmname.qcow2 /mnt/evidence/vmname.qcow2
# Resume VM if needed
virsh resume vmname

Document the VM configuration as well (e.g. virsh dumpxml vmname).

Log and configuration evidence

System logs

On Linux, key logs are usually under /var/log. Collect at least:

Example: 

mkdir -p /mnt/evidence/var_log
cp -a /var/log/* /mnt/evidence/var_log/

Also capture log rotation configs: 

cp -a /etc/logrotate.conf /etc/logrotate.d /mnt/evidence/logrotate/

Application and service logs

This depends on the system; common locations:

Copy relevant directories recursively, preserving permissions and timestamps: 

cp -a /var/log/nginx /mnt/evidence/var_log_nginx 2>/dev/null || true
cp -a /var/log/apache2 /mnt/evidence/var_log_apache2 2>/dev/null || true
cp -a /var/log/mysql /mnt/evidence/var_log_mysql 2>/dev/null || true

Configuration files

Configurations are crucial for reconstructing how a system was supposed to behave vs how it was misused.

Common targets: 

  cp -a /etc /mnt/evidence/etc_full

Remote and off-host evidence

Modern environments generate lots of evidence outside the compromised machine.

Networking and security devices

Coordinate with relevant teams to acquire:

Cloud and orchestration logs

For Linux systems in cloud or containerized environments, evidence may include:

These are usually viewed and exported via provider tools or APIs (not covered here in depth). The key for this chapter: treat them as first-class evidence, with the same hash and chain-of-custody discipline.

Minimizing contamination and anti-forensic defenses

Reducing your footprint

Every action you take may:

Mitigations:

Handling anti-forensics and rootkits

If you suspect:

Then:

Example: analyzing a mounted image from a clean system: 

# On a forensic workstation
mount -o ro,loop,show_sys_files,relatime disk-sda.img /mnt/image
# Now use host tools (not the suspect system's) to inspect
ls -al /mnt/image/bin

Packaging and transferring evidence

Structuring evidence directories

Use a logical structure, for example: 

/mnt/evidence/
  host-web01/
    metadata/
      hostinfo.txt
      acquisition-notes.txt
    live/
      date.txt
      ps_aux.txt
      ss_tcp_all.txt
      ...
    logs/
      var_log/
      app_logs/
    configs/
      etc_full/
    images/
      disk-sda.img
      disk-sda.img.sha256
      memdump.lime
      memdump.lime.sha256

This makes it easier for analysts to navigate later.

Secure transfer and storage

When moving evidence:

Example: 

# On source
tar czf - host-web01 | gpg --encrypt --recipient forensics@example.com \
  > host-web01.tar.gz.gpg
# Transfer
scp host-web01.tar.gz.gpg forensics@example.com:/evidence/incoming/
# On destination, verify and re-hash
sha256sum host-web01.tar.gz.gpg > host-web01.tar.gz.gpg.sha256

Implement access control on the evidence repository (least privilege), and configure backups to avoid accidental loss without allowing uncontrolled modification.

Practicing evidence collection

To become effective, practice in non-production environments:

Over time, refine:

That combination of tooling and discipline is what turns ad-hoc data gathering into reliable, defensible forensic evidence collection.

Previous
Up
Next

Views: 29

Comments

Please login to add a comment.

Don't have an account? Register now!

© 2025 Kahibaro

Visitors: 1669 / 24h

Impressum / Imprint

Datenschutz / Data Policy