Analyzing suspicious processes

Goals and mindset for process analysis

When you see a suspicious process, your goals are to:

• Identify what it is and what started it.
• Understand what it is doing now.
• Decide whether it’s malicious, misconfigured, or benign.
• Preserve enough information for later investigation or evidence.
• Contain or terminate it safely without losing critical data.

In incident response, you usually observe first, change later:

• Collect as much volatile and semi‑volatile data as you reasonably can (commands, network connections, memory if possible).
• Avoid rebooting or killing processes until you’ve captured the essentials—unless there is immediate risk (e.g., ransomware actively encrypting).

This chapter assumes you already have:

• A basic incident response workflow.
• Familiarity with common monitoring tools (ps, top, /var/log, etc.).

Here we focus specifically on how to interrogate suspicious processes.

Recognizing suspicious processes

You rarely start from a blank slate. Typical triggers:

• Host‑based or network security alerts.
• Unusual resource use (CPU, RAM, disk, network).
• Unexpected listening ports or outbound connections.
• New binaries in odd locations.
• User reports ("system is slow", "fans at 100%", "I never ran this").

When scanning processes, red flags include:

• Weird or random names: kworker/0:0H, systemd-udevd clones in user home dirs, dbus-daemon2, bashd, kthreadd, crond in non‑standard locations.
• Command paths that don’t match the name: a process named sshd executing from /tmp/.X11-unix/sshd.
• Long or obfuscated command lines: huge base64 blobs, heavily escaped shell commands, curl|bash, wget|sh, PowerShell equivalents in cross‑platform systems.
• Unexpected parent/child relationships: apache2 spawning bash that spawns ssh or nc.
• Strange user context: a system daemon running as a regular user, or interactive tools (bash, python, perl) running as www-data, nginx, mysql.
• High privilege behavior: processes running as root from /tmp, /dev/shm, user home directories.
• Persistence behavior: same suspicious process reappearing after being killed or after reboot.

Capturing a process snapshot

Before changing anything, take a quick snapshot of what’s running and what each suspicious process is doing. This is often your only chance if an attacker notices your presence.

Baseline process listing

Use ps with extensive columns to capture a point‑in‑time process list:

ps auxww > /root/ps_snapshot_$(date +%F_%H%M%S).txt

Options explained briefly:

• a – all users’ processes with terminals.
• u – show owner and CPU/mem usage.
• x – include processes without a controlling terminal (daemons).
• ww – don’t truncate wide command lines.

For tree view, which is vital to understand parent–child relationships:

ps axfww -o pid,ppid,uid,gid,cmd > /root/ps_tree_$(date +%F_%H%M%S).txt

Look for:

• Unexpected processes with PPID 1 (parent is init/systemd).
• Shells, interpreters, or compilers under web servers or other daemons.

Using `pstree` for hierarchy

pstree is useful for visual inspection:

# Entire tree, with PIDs and user names
pstree -alp > /root/pstree_$(date +%F_%H%M%S).txt
# Filter by a specific suspicious PID
pstree -alp <PID>

This helps answer:

• Who launched this?
• Are there multiple related processes?
• Is there a process chain (e.g., sshd -> bash -> python -> nc)?

Take targeted snapshots

For each suspicious process <PID>, immediately save:

• Command and environment:

  # Process info
  ps -p <PID> -o pid,ppid,user,group,start,time,cmd >> /root/proc_<PID>.txt
  # Environment if readable
  tr '\0' '\n' < /proc/<PID>/environ >> /root/proc_<PID>_env.txt 2>/dev/null

• Current working directory:

  ls -al /proc/<PID>/cwd >> /root/proc_<PID>_cwd.txt

• Open file descriptors:

  ls -l /proc/<PID>/fd >> /root/proc_<PID>_fd.txt

• Memory maps (for advanced or if memory dumping isn’t possible):

  cat /proc/<PID>/maps > /root/proc_<PID>_maps.txt

These /proc paths are volatile but extremely informative while the process lives.

Inspecting a single suspicious process

Once you have candidate PIDs, dive into each one methodically.

Verify the executable and its location

Use ls and /proc/<PID>/exe:

# What binary is this process actually executing?
ls -l /proc/<PID>/exe
# Where is it on disk?
realpath /proc/<PID>/exe

Questions to ask:

• Does the path align with the process name?
• sshd expected in /usr/sbin/sshd, not /tmp/sshd.
• Is the binary in a writable location (/tmp, /var/tmp, /dev/shm, user home)?
• Is it owned by an unexpected user?
• Is it recently modified?:

  stat -c 'Owner:%U Group:%G Size:%s mtime:%y' "$(realpath /proc/<PID>/exe)"

If the binary is missing but the process still runs, /proc/<PID>/exe will be dangling ((deleted)); this is common for malware trying to hide:

ls -l /proc/<PID>/exe
# Example output:
# lr-x------ 1 root root 0 ... /proc/1234/exe -> /tmp/.xyz (deleted)

In that case, use cp /proc/<PID>/exe (see below) to preserve the image.

Collect a copy of the executable image

If allowed by policy, copy the binary for offline analysis:

cp /proc/<PID>/exe /root/evidence/exe_<PID>_$(date +%F_%H%M%S)
sha256sum /root/evidence/exe_<PID>_* >> /root/evidence/checksums.txt

If /proc/<PID>/exe is a deleted binary, this is sometimes the only way to recover it.

Examine the command line

The command line shows how the process was started:

tr '\0' ' ' < /proc/<PID>/cmdline; echo

Look for:

• Execution from temporary directories:
• bash /tmp/update.sh
• One‑liners from curl, wget, python, perl, php:
• bash -c "curl http://… | sh"
• Shell features: backticks, $(...), heavy use of eval, base64 decode, etc.

Preserve it as evidence:

tr '\0' ' ' < /proc/<PID>/cmdline >> /root/proc_<PID>_cmd.txt

Environment variables

Environment can reveal:

• Stolen credentials (hard‑coded tokens, AWS keys).
• Proxy/bypass settings (http_proxy, https_proxy, LD_PRELOAD, LD_LIBRARY_PATH).
• Indicators related to malware campaigns.

tr '\0' '\n' < /proc/<PID>/environ | sort >> /root/proc_<PID>_env.txt 2>/dev/null

Sensitive info may appear here, so handle carefully and store securely.

Current working directory and associated files

The working directory often points to where the process or its payload resides:

ls -al /proc/<PID>/cwd
realpath /proc/<PID>/cwd
ls -al "$(realpath /proc/<PID>/cwd)" > /root/proc_<PID>_cwd_list.txt

Suspicious if:

• CWD is /tmp, /dev/shm, or a user’s download folder.
• Contains executables or scripts with odd names or extensions.

Inspecting open files and network activity

Understanding what a process has open is crucial for spotting exfiltration, keylogging, crypto‑mining, or lateral movement.

`lsof` overview

lsof (“list open files”) is one of the most useful tools:

# All open files for a PID
lsof -p <PID> > /root/proc_<PID>_lsof.txt
# Only network‑related
lsof -Pan -p <PID> -i
# All listening sockets system‑wide
lsof -Pan -iTCP -sTCP:LISTEN

Pitfalls:

• lsof might not be installed everywhere; install from your distro packages if policy permits.
• Running it as non‑root may show incomplete information.

Network connections via `/proc` or `ss`

Brief, low‑level methods:

# Human-friendly socket info
ss -pant | grep <PID>
# Or use 'netstat' on older systems
netstat -plant | grep <PID>

Suspicious signs:

• Outbound connections to unusual or geo‑unexpected IPs and ports.
• Connections to known command‑and‑control or crypto‑mining pools.
• Many short‑lived outbound connections (scanning or brute forcing).

For encrypted connections, you may not see payloads, but you still get:

• Destination IP/port.
• Connection state (e.g., ESTAB, SYN_SENT).
• Local port (may reveal a reverse shell, e.g., outbound to 443 from a shell).

Open files and pipes

From lsof -p <PID> or /proc/<PID>/fd, note:

• Files in:
• /etc (configs being altered).
• /var/log (log tampering).
• /home directories (data theft).
• /tmp, /dev/shm (payload staging).
• Pipes and sockets:
• Names like pipe:[12345] between suspicious PIDs.
• anon_inode:[eventfd], inotify watchers on many files (may indicate data monitoring).
• Deleted but still open files:
• lsof will show (deleted) entries; contents can still be read via /proc/<PID>/fd/<n>.

Example to dump an open deleted file:

# Identify fd number from lsof output, e.g., 5u REG /tmp/.x (deleted)
cp /proc/<PID>/fd/5 /root/evidence/deleted_<PID>_fd5

CPU, memory, and behavior patterns

Resource use patterns can indicate certain malware families or activities.

CPU and memory usage

Use top/htop interactively, or ps for snapshots:

ps -p <PID> -o pid,user,%cpu,%mem,etime,cmd

Clues:

• High, sustained CPU from unknown processes (crypto‑miners, brute‑forcers).
• Rapid growth in memory usage (memory leak, DoS, or data aggregation).
• Long etime (elapsed time) for processes you don’t recognize (they may have persisted quietly).

Thread and child process activity

Many advanced threats spawn numerous threads or children:

# Count threads
ls -1 /proc/<PID>/task | wc -l
# View tasks
ls -1 /proc/<PID>/task

Extremely high thread counts can be a sign of:

• DDoS bots.
• Scanners using many threads.

Use ps to see children:

ps --ppid <PID> -o pid,user,%cpu,%mem,cmd

Correlating processes with users and logins

Identify which user context and login session a suspicious process is tied to.

Basic user mapping

From ps:

ps -p <PID> -o pid,user,group,cmd

Look up the user in /etc/passwd or with id:

id <username>

Questions:

• Is this a system user (www-data, nginx, mysql) running tools they normally wouldn’t (e.g., bash, ssh, scp)?
• Is the account interactive when it should be service‑only?

TTY and login sessions

If the process is interactive (shell, editor), see if it is attached to a terminal:

ps -p <PID> -o tty,pid,user,cmd

For interactive logins, correlate with:

who
w
last -n 20

This helps you answer:

• Did someone log in legitimately and then run this?
• Was the login from an unusual IP?

Identifying persistence linked to processes

A suspicious process might be part of a persistence mechanism. Use your process as a starting point to hunt:

• Parent process:
  If parent is systemd or a system service, check the relevant unit file later in the persistence analysis (covered by other chapters).
• Cron and timers:
  If ppid is crond or a timer, note which job file or timer triggered it.
• Init scripts or custom services:
  Names that map to /etc/init.d/, /etc/systemd/system/ units, or user‑level services under ~/.config/systemd/user/.

From the process command line or tree, jot down:

• Service names.
• Script paths.
• Cron paths (/etc/cron.*, user crontabs).

You will analyze these in depth as part of persistence and configuration review, not necessarily at process‑time.

Handling kernel‑level or stealth processes (rootkits)

Some advanced threats hide processes from userland tools.

Indicators of possible rootkit influence:

• ps, top, ls, netstat, ss, or lsof output doesn’t match each other.
• A known PID appears in /proc but not in ps output, or vice versa.
• System call anomalies or unexplained failures.

Basic cross‑checks:

• Compare /proc vs ps:

  # Count numeric dirs in /proc (rough process count)
  ls /proc | grep '^[0-9]\+$' | wc -l
  # Compare with: 
  ps ax | wc -l

• Use statically linked or trusted binaries from a separate media (e.g., USB toolkit) to reduce the chance of tampering.

In deeper cases, you may:

• Use forensic or rootkit‑detection tools like chkrootkit, rkhunter, or more advanced frameworks (as allowed by your environment).
• Rely on remote memory acquisition and offline analysis.

Those topics are typically covered in specialized forensic tooling chapters; here, recognize that when process listings are inconsistent, your trust in local tools must decrease.

Deciding what to do with a suspicious process

Once you understand a process reasonably well, choose how to act:

Preserve, then contain

Trigger checklist before termination (if time permits and risk is acceptable):

1. Captured:
• ps outputs and process tree.
• /proc/<PID>/cmdline, environ, cwd, fd, maps.
• Network connections from ss/lsof.
• Executable copy from /proc/<PID>/exe (if allowed).
2. Correlated:
• Parent PID and its process info.
• User account, login session, and basic logs around start time.
3. Checked:
• Any obvious persistence (systemd units, cron jobs, startup scripts).

Terminating or isolating

If the process is actively harmful (e.g., encryption, data exfiltration):

• Prefer isolation over blind killing:
• Temporarily block outbound network at firewall level.
• Disconnect the host from the network (if allowed).
• If you must kill:
• Try graceful kill first:

    kill <PID>        # SIGTERM

• If no effect and immediate danger:

    kill -9 <PID>     # SIGKILL (irreversible, bypasses cleanup)

• Consider that killing a process may:
• Trigger a watchdog that restarts it.
• Destroy memory‑resident evidence.
• Disrupt legitimate services if attribution was mistaken.

Document:

• Time and method of termination.
• Any immediate behavioral change (CPU usage drop, network connections closed, log entries).

Practical triage workflow example

A minimal, repeatable workflow you can apply under pressure:

1. Identify candidates
• ps auxww and ps axfww to locate odd processes.
2. Pick one suspicious PID
• Criteria: unknown name, odd location, high CPU, strange parent.
3. Snapshot everything quickly
• ps -p <PID> -o pid,ppid,user,%cpu,%mem,etime,cmd
• /proc/<PID>/{cmdline,environ,cwd,fd,maps} to files in /root/evidence.
• lsof -p <PID> and ss -pant | grep <PID>.
• Copy /proc/<PID>/exe if permitted.
4. Analyze context
• Confirm binary path and owner.
• Examine environment and working directory.
• Check open network connections and files.
• Map to user and login session (who, last).
5. Check for relatives
• pstree -alp <PID> and ps --ppid <PID>.
• Investigate parent PID similarly if suspicious.
6. Decide action
• If benign ⇒ document and move on.
• If malicious/suspicious ⇒ coordinate containment, then terminate after evidence captured.
7. Log everything
• Commands run.
• Files collected (with hashes).
• Observations and timestamps.

This systematic approach helps you remain consistent and defensible in incident response, even under time pressure.