File integrity monitoring

Why File Integrity Monitoring Matters

File integrity monitoring (FIM) is about answering two questions reliably:

• What changed?
• Who/what changed it and when?

It’s a core hardening control because many attacks involve:

• Modifying binaries (/bin/, /usr/bin/), configs (/etc/), or scripts
• Dropping backdoors or web shells
• Changing permissions/ownership to persist access
• Tampering with logs to hide traces

FIM gives you a way to detect such changes, validate them, and respond.

Key use-cases:

• Detecting web shell uploads on a web server
• Noticing unauthorized changes to /etc/sudoers or SSH config
• Spotting unexpected cron jobs
• Compliance requirements (PCI-DSS, HIPAA, etc.)

What to Monitor

You cannot (and usually should not) monitor the entire filesystem with the same intensity. Effective FIM is about scoping.

Typical high-value targets:

• System binaries:
• /bin, /sbin, /usr/bin, /usr/sbin
• Critical configuration:
• /etc (but often with exclusions for frequent-changers)
• /etc/passwd, /etc/shadow, /etc/group, /etc/sudoers
• Network and service configs (e.g. /etc/ssh/sshd_config, web server configs)
• Boot and kernel related:
• /boot
• Module directories like /lib/modules
• Task schedulers and persistence mechanisms:
• /etc/cron.*, /var/spool/cron
• systemd unit overrides in /etc/systemd/system
• Web application areas:
• Document roots like /var/www, but often tuned to ignore logs/cache/uploads as needed
• Security tools’ configs:
• IDS/IPS, firewall, and FIM configs themselves

Common exclusion patterns (to avoid noise and overhead):

• Log directories: /var/log/**
• Temporary directories: /tmp/, /var/tmp/, runtime dirs like /run/**
• Package manager caches: /var/cache/, /var/lib/apt/lists/, etc.
• VCS metadata (like .git/), app cache directories

The strategy is to:

1. Start with a small, clearly defined scope.
2. Run in report mode, tune out noisy paths.
3. Only then expand if genuinely needed.

Core FIM Techniques and Concepts

Checksums and Hashes

At the core of FIM is the idea of taking a “snapshot” of file attributes and comparing later snapshots.

Commonly recorded attributes:

• Path, size, timestamps (mtime, ctime)
• Ownership (UID/GID), permissions, SELinux context
• Cryptographic hash (e.g. SHA-256)

Hashes allow you to determine content changes rather than just metadata changes.

• Store: $h_1 = \text{hash}(\text{file})$
• Later: $h_2 = \text{hash}(\text{file})$
• If $h_1 \neq h_2$, the file content changed.

Use strong hashes (SHA-256, SHA-512) rather than weak ones (MD5, SHA-1).

Realtime vs Periodic Monitoring

Two major modes:

• Periodic (scheduled)
• A job (cron, systemd timer) runs a check at intervals.
• Lower overhead, simpler, but changes might be detected after the fact.
• Good for static areas like /usr/bin, /boot.
• Realtime (event-based)
• Kernel notifies FIM via inotify/fanotify or audit subsystem.
• Changes are seen almost immediately.
• Higher complexity and overhead, best for high-risk areas (e.g. /etc, web roots).

Many tools support both; you typically mix them.

Baseline Creation and Trust

FIM is only as trustworthy as its baseline — the initial snapshot.

Recommended baseline process:

1. Install/configure the system.
2. Patch and harden it to your desired “known-good” state.
3. Ensure the system is clean (e.g. initial integrity checks, malware scan if needed).
4. Generate the initial FIM database or baseline.
5. Secure the database (ideally off-host or with strong access controls).

If the system is already suspect, creating a baseline on it is risky; it might simply “bless” compromised files.

Common FIM Tools and Approaches

This section focuses on typical open-source solutions. Commercial/enterprise FIM products build on the same principles but add dashboards, centralization, and compliance reporting.

Tripwire (Conceptual Overview)

Tripwire was an early, influential FIM tool:

• Maintains a database of file attributes and hashes.
• Runs checks and reports diffs.
• Uses cryptographic signatures to protect its database and config.

On modern systems, it’s less common than other tools (e.g. AIDE) but the model is similar: initialize → check → investigate changes → update baseline when appropriate.

AIDE (Advanced Intrusion Detection Environment)

AIDE is widely used as a straightforward, host-based FIM.

Typical Workflow

1. Install aide via your package manager.
2. Configure what to monitor in /etc/aide/aide.conf (paths, rules).
3. Initialize the database:

   sudo aide --init
   sudo mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db

4. Schedule regular checks (via cron or systemd timer).
5. Review reports, investigate unexpected changes, and update the database when legitimate changes occur:

   sudo aide --check
   sudo aide --update
   # then move the new db into place after review

Rule Concept

AIDE uses rule sets specifying which attributes are checked, e.g.:

• p – permissions
• i – inode
• n – number of links
• u – user
• g – group
• s – size
• m – mtime
• a – atime
• c – ctime
• acl – ACLs
• sha256, sha512 – hash algorithms

Example configuration fragment:

# Define a rule set
BinRule = p+i+n+u+g+s+m+c+sha256
# Apply it
/bin    BinRule
/sbin   BinRule
/usr/bin BinRule
/usr/sbin BinRule
# Example for /etc with a more detailed rule
/etc    p+i+n+u+g+s+m+c+acl+sha256

You’d then add exclusions for noisy paths (e.g. /etc/mtab on some systems).

Auditd-Based FIM

The Linux audit subsystem (auditd) can be used to monitor file-related syscalls:

• You define rules that watch paths or syscalls.
• When something changes, an audit record is generated.
• Tools (like ausearch, aureport, or SIEM agents) consume and analyze the logs.

Example rule concept (not full syntax coverage):

• Watch a file for write/attribute changes, logged with a custom key:

  -w /etc/shadow -p wa -k shadow-watch

• -w: watch path
• -p: permissions to watch (r, w, x, a for attribute change)
• -k: tag events with a key for easier searching

Audit-based FIM is powerful because it can tell you:

• Which process wrote the file
• Which user initiated the change
• The syscall, PID, and other context

It’s best suited for realtime detection and forensic context; you can combine it with hash-based tools for content comparison.

OSSEC / Wazuh (Agent-Based HIDS with FIM)

OSSEC and its fork Wazuh include FIM as part of a full host-based intrusion detection system (HIDS):

• Agents run on endpoints (Linux, Windows, etc.).
• They watch file changes and send events to a manager.
• The manager correlates events, raises alerts, and can perform active responses.

Key points for FIM:

• Define monitored directories and file types in the agent config.
• Use both realtime and periodic checks.
• Forward alerts to your SIEM or notification channel.

This approach is well-suited for environments with many servers where you want centralized FIM and richer correlation.

Designing a Practical FIM Policy

Step 1: Define Objectives

Clarify what you care about:

• Are you mainly worried about:
• Rootkits and system binary tampering?
• Web-facing content modifications?
• Insider changes to configs?
• Compliance (e.g., “monitor all cardholder environment files”)?

Objectives determine:

• Which directories to watch
• How often to check
• What triggers alerts vs. just logs

Step 2: Scope and Performance Considerations

Overly broad FIM can:

• Hurt performance
• Generate too many alerts (alert fatigue)
• Become unmanageable

Guidelines:

• Start narrow: critical system + security configs.
• Use periodic checks for large, mostly-static trees.
• Use realtime for high-risk, smaller sets (like /etc, web roots).
• Exclude:
• Logs, temp, caches, large frequently-changing data sets.

Always test FIM impact on non-production or low-risk systems first.

Step 3: Alerting and Integration

FIM is only useful if someone acts on its findings.

Typical integration patterns:

• Send reports via email (small deployments).
• Forward events to a SIEM (e.g. via syslog/json).
• Integrate with chat-alert systems (Slack, Teams, etc.).
• Store baselines and logs off-host, or on hardened, append-only storage.

Control noise:

• Classify changes:
• Critical: system binaries, auth files, security configs.
• Important: application configs.
• Informational: non-security-sensitive files.
• Tune alerts to focus on critical/important, while still logging everything of potential interest.

Step 4: Baseline Maintenance and Change Management

FIM must align with your legitimate change process:

• Before planned changes (patching, deployment):
• Optionally disable high-noise alerts or accept them as “expected”.
• After changes:
• Run a check, review diffs for unexpected items.
• If everything is OK, update the FIM baseline.

Never blindly update baselines:

• Always skim the report:
• Did any unexpected file change?
• Are there newly created files in suspicious places?
• For high-security systems, consider peer review of FIM changes.

Protecting the FIM Itself

An attacker who gains control of the system might try to:

• Disable FIM service or delete logs.
• Modify config to remove critical paths from monitoring.
• Tamper with the FIM database to hide past changes.
• Block outbound alerts.

Countermeasures:

• Store baselines and configs on read-only or restricted media where possible.
• Export FIM reports and audit logs off-host quickly (central log server / SIEM).
• Monitor integrity of FIM-related files themselves:
• FIM configs
• FIM binaries
• Use service monitoring (e.g. systemd, external monitoring) to ensure the FIM agent is running.
• On high-sensitivity systems, consider:
• Immutable attributes (chattr +i on some files, where appropriate).
• Read-only root partitions or verified boot (covered elsewhere in hardening topics).

Interpreting and Responding to FIM Events

Detection without response doesn’t help. You should have a basic playbook.

Triage Questions

When a change is detected:

1. Is the change expected?
• Was there a deployment, patch, config update?
2. What changed exactly?
• Diff of config or script, changed permissions, new binary?
3. Who/what changed it?
• Correlate with:
• Audit logs
• Shell history
• Configuration management runs
4. Is there potential impact?
• Does this enable privilege escalation, backdoor access, or data exfiltration?

Typical Response Actions

Depending on severity:

• Document the change and update baseline (for legitimate modifications).
• Revert the file from:
• Version control (e.g. Git)
• Configuration management (e.g. Ansible, Puppet)
• Known-good backups
• Isolate the host (network segmentation) if you suspect compromise.
• Launch a broader incident response process:
• Check for additional tampering.
• Examine process list, network connections, and log files.
• Preserve evidence.

FIM is often the first indicator of hidden compromise; treat suspicious events seriously, even if everything else “appears fine”.

Best Practices Summary

• Monitor critical system and security files by default (/etc, system binaries, boot, auth-related files).
• Use strong hashes like SHA-256 and protect FIM baselines and configs.
• Combine periodic scans for broad coverage with realtime monitoring for high-risk locations.
• Integrate FIM with your logging and alerting pipelines, and tune noise aggressively.
• Align FIM baselines with your change management process; never auto-accept changes.
• Regularly test that FIM still works:
• Confirm alerts when you make a controlled test change.
• Check that logs reach your central system.
• Treat unexpected FIM alerts as potential security incidents, not mere anomalies.

Properly implemented, file integrity monitoring becomes a powerful complement to your other hardening controls, giving you early visibility into unauthorized changes and helping you maintain trust in your systems over time.