Table of Contents
Role of `/var/log` in Traditional Logging
On most Linux systems, /var/log is the central directory where traditional (non-journald) log files are stored. Even on systemd-based systems that primarily use journald, many services still write text logs here, either directly or via a classic syslog daemon (rsyslog, syslog-ng, etc.).
In this chapter, the focus is on:
- The structure and purpose of key files and subdirectories in
/var/log - How to read and interpret common logs
- How log rotation interacts with
/var/log - Practical workflows for troubleshooting using traditional logs
General concepts of logging and auditing, and systemd’s journal, are covered in their own chapters; here we stay at the level of concrete files.
Layout and Conventions in `/var/log`
While exact contents vary by distribution, some conventions are common:
- System-wide logs are typically plain text files under
/var/log - Application-specific logs often live under
/var/log/<application>/ - Old logs are rotated with suffixes like
.1,.2.gz, etc. - Many logs are owned by
rootand requiresudoto view
Common file naming patterns:
- Base name: type or source of log (
syslog,messages,auth.log) - Date/number suffixes: rotated archives (
auth.log.1,auth.log.2.gz) .gz: compressed log archives
Example listing:
$ ls /var/log
alternatives.log auth.log dmesg kern.log
apt/ boot.log dpkg.log syslog
journal/ lastlog wtmp btmp
nginx/ samba/ Xorg.0.log ...Many of these are created and managed by package-specific configuration; not every system will have all of them.
Core System Logs
`/var/log/syslog` and `/var/log/messages`
These are general-purpose system logs used by many distributions, but there are differences:
- Debian/Ubuntu: primarily use
/var/log/syslog - RHEL/CentOS/Fedora: primarily use
/var/log/messages
Both contain a large variety of messages:
- Kernel messages (unless separated into
kern.log) - System services (if they log via syslog)
- General informational and error messages
Example lines from /var/log/syslog:
Jan 10 09:12:34 host1 NetworkManager[742]: <info> [1641805954.1234] device (wlp2s0): state change: ...
Jan 10 09:12:35 host1 CRON[10234]: (root) CMD (/usr/local/bin/backup.sh)
Jan 10 09:12:36 host1 kernel: [12345.678901] usb 1-1: new high-speed USB device number 5 using xhci_hcdKey columns to recognize:
- Timestamp:
Jan 10 09:12:34 - Hostname:
host1 - Process and PID:
NetworkManager[742]: - Message: the rest of the line
Typical uses:
- General troubleshooting (“what happened around 09:15?”)
- Checking if services using syslog are logging as expected
- Correlating events from multiple services by timestamp
Basic viewing:
sudo less /var/log/syslogsudo tail -f /var/log/syslog(follow in real time)
`/var/log/kern.log`
On some distributions (especially Debian-based), kernel messages are separated to /var/log/kern.log in addition to being visible via dmesg.
Typical content:
- Hardware detection and errors
- Driver messages
- Filesystem errors
- Network stack diagnostics
Example entries:
Jan 10 09:12:36 host1 kernel: [12345.678901] ata1.00: status: { DRDY ERR }
Jan 10 09:12:36 host1 kernel: [12345.678902] e1000e 0000:00:19.0 eth0: Link is Up 1000 Mbps, Full DuplexUse cases:
- Investigating hardware issues (disks, NICs, USB)
- Diagnosing kernel-level network problems
- Spotting filesystem or block device errors early
Boot and Startup Logs
`/var/log/boot.log`
On systems using bootlogd or similar mechanisms, /var/log/boot.log stores messages from the early boot sequence.
Features:
- Shows services starting during boot
- Captures messages you might have seen flash by on the console
Example:
* Starting system log daemon rsyslogd [ OK ]
* Starting OpenBSD Secure Shell server sshd [ OK ]
* Starting web server apache2 [fail]Use cases:
- Debugging services that fail only during boot
- Comparing successful vs failed boot sequences
`/var/log/dmesg` (if present)
Some systems store the boot-time kernel ring buffer into /var/log/dmesg. This is a snapshot similar to the output of dmesg shortly after boot.
Useful for:
- Reviewing early hardware initialization messages
- Diagnosing boot-time driver problems
Authentication and Security Logs
`/var/log/auth.log` (Debian/Ubuntu) and `/var/log/secure` (RHEL/Fedora)
These files track authentication-related events and security-sensitive actions.
They typically include:
- SSH logins and failures
sudousage- Local logins (TTY, GUI display manager, etc.)
- PAM (Pluggable Authentication Modules) messages
Example (auth.log):
Jan 10 10:01:23 host1 sshd[2123]: Failed password for invalid user admin from 203.0.113.5 port 45678 ssh2
Jan 10 10:01:26 host1 sshd[2123]: Accepted password for alice from 203.0.113.5 port 45678 ssh2
Jan 10 10:02:00 host1 sudo: alice : TTY=pts/0 ; PWD=/home/alice ; USER=root ; COMMAND=/bin/systemctl restart nginxUse cases:
- Detecting brute-force login attempts (lots of failed SSH logins)
- Auditing who used
sudo, when, and for what command - Investigating suspicious logins or account lockouts
Practical commands:
- Last 100 authentication events:
sudo tail -n 100 /var/log/auth.log- See failed SSH attempts:
sudo grep "Failed password" /var/log/auth.log`/var/log/faillog` and related files
Some distributions use database-like log files to track failed logins:
/var/log/faillog: failed login attempts per user/var/log/tallylog(on some systems): similar purpose
These are typically not plain text. Use commands like faillog to read them:
sudo faillog -aLogin History Logs: `wtmp`, `btmp`, `lastlog`
These special files keep historical data about login activity. They’re binary and require specific tools.
`/var/log/wtmp`
Tracks user login and logout events, plus system boots and shutdowns.
To read it:
last(no arguments) reads/var/log/wtmpby default:
$ last
alice pts/0 203.0.113.5 Mon Jan 10 10:01 still logged in
bob pts/1 203.0.113.10 Mon Jan 10 09:30 - 10:00 (00:30)
reboot system boot 5.10.0-10-amd64 Mon Jan 10 09:00 still runningUses:
- Auditing who logged in and when
- Checking system reboot history
`/var/log/btmp`
Tracks failed login attempts. It’s also binary.
To read it:
sudo lastbExample:
root ssh:notty 203.0.113.5 Mon Jan 10 10:01 - 10:01 (00:00)Useful for:
- Seeing which accounts attackers are trying to guess
- Estimating the volume of failed logins
`/var/log/lastlog`
Stores the most recent login time for each user account.
To read:
sudo lastlogExample:
Username Port From Latest
root pts/0 203.0.113.5 Mon Jan 10 10:01:26 +0000 2025
alice pts/1 192.0.2.15 Sun Jan 9 18:12:10 +0000 2025Use cases:
- Spotting dormant accounts (never logged in or very old last login)
- Simple account activity overview
Package and System Management Logs
`/var/log/apt/` (Debian/Ubuntu)
APT’s subdirectory with logs such as:
history.log: what packages were installed, removed, or upgradedterm.log: terminal output duringaptoperations
Example history.log snippet:
Start-Date: 2025-01-10 10:20:00
Commandline: apt-get install nginx
Install: nginx:amd64 (1.18.0-0ubuntu1)
End-Date: 2025-01-10 10:20:05Useful for:
- Tracing back which change broke something
- Auditing what software was installed when
`/var/log/dpkg.log` (Debian/Ubuntu)
Lower-level log for the dpkg package manager. Contains detailed records of package installation/removal/configure operations.
Example:
2025-01-10 10:20:01 install nginx:amd64 <none> 1.18.0-0ubuntu1
2025-01-10 10:20:02 status installed nginx:amd64 1.18.0-0ubuntu1`/var/log/yum.log` or `/var/log/dnf.log` (RHEL/Fedora-based)
Equivalent logs for RPM-based systems, tracking actions by yum or dnf.
Use:
- Understanding when a package or update was applied
- Rolling back or reproducing package states (together with other tools)
Service-Specific Logs Under `/var/log`
Many services create their own subdirectories under /var/log. Some common examples:
Web Server Logs
Apache HTTP Server: `/var/log/apache2/` (Debian/Ubuntu) or `/var/log/httpd/` (RHEL/Fedora)
Typical files:
access.log: every HTTP request (if configured)error.log: errors, warnings, and startup/shutdown messages
Example access.log line:
203.0.113.10 - - [10/Jan/2025:10:30:45 +0000] "GET /index.html HTTP/1.1" 200 1024 "-" "Mozilla/5.0 ..."Uses:
- Troubleshooting web issues
- Basic traffic analysis and security monitoring
Nginx: `/var/log/nginx/`
Similar structure:
access.logerror.log
Application and Service Logs
Other typical service directories:
/var/log/samba//var/log/mysql/or/var/log/mariadb//var/log/cronorcronentries insidesyslog/messages/var/log/Xorg.0.logfor X server display issues
Each application’s own documentation or configuration (often in /etc/) will define the exact log file names and formats.
Log Rotation and Archived Logs
Traditional text logs in /var/log can grow quickly. logrotate (described in another chapter) manages their size and retention, but you should recognize rotated logs:
Common patterns:
- Current active file:
syslog - Previous file:
syslog.1 - Older compressed archives:
syslog.2.gz,syslog.3.gz, ...
Important points:
- Use
zless,zcat, orzgrepto read.gzlogs without manually decompressing:
sudo zless /var/log/syslog.2.gz
sudo zgrep "nginx" /var/log/syslog.3.gz- Rotation is usually configured per-log in
/etc/logrotate.d/and a main config file in/etc/logrotate.conf.
Typical workflow:
- Look at current log:
sudo tail -n 50 /var/log/auth.log- If the time range you need is older, move to rotated logs:
sudo zgrep "Jan 9" /var/log/auth.log.1.gzPractical Troubleshooting Workflows with `/var/log`
Here are some focused examples of how /var/log is used in day-to-day work.
Checking for Failed SSH Logins
# Recent failed SSH logins (Debian/Ubuntu)
sudo grep "Failed password" /var/log/auth.log | tail
# On RHEL/Fedora
sudo grep "Failed password" /var/log/secure | tailFinding Why a Service Failed to Start (Traditional Logging)
If a service logs via syslog and not only via systemd’s journal:
sudo grep nginx /var/log/syslog | tail
# or
sudo grep nginx /var/log/messages | tail
Then pivot to /var/log/nginx/error.log for more details.
Tracing a System Change to a Package Update
# See what was changed around the time of the problem (Debian/Ubuntu)
sudo less /var/log/apt/history.log
sudo less /var/log/dpkg.log
# On RHEL/Fedora
sudo less /var/log/dnf.logInvestigating a Suspected Hardware Issue
# Check kernel messages related to disks, filesystems, etc.
sudo less /var/log/kern.log # if present
# or search within syslog/messages
sudo grep -i "error" /var/log/syslog | tailLook for patterns like I/O errors, resets, or repeated warnings.
Permissions and Access Considerations
Because logs often contain sensitive information, many files in /var/log are:
- Owned by
root - Not readable by normal users
Typical permissions:
-rw-r----- 1 root adm 12345 Jan 10 10:35 auth.log
-rw-r----- 1 root adm 67890 Jan 10 10:35 syslogImplications:
- Use
sudowhen reading sensitive logs:
sudo less /var/log/auth.log- Be cautious when copying or sharing log files—redact IPs, usernames, or other sensitive data when necessary.
Summary
Traditional logs in /var/log remain crucial even on modern, systemd-based systems:
- System-wide diagnostic logs:
syslog,messages,kern.log,boot.log - Security and authentication:
auth.logorsecure,wtmp,btmp,lastlog - Package management history:
apt/,dpkg.log,dnf.log,yum.log - Application-specific logs: directories like
apache2/,nginx/,samba/
Being effective with /var/log is about:
- Knowing where to look for a given type of event
- Understanding basic log formats (timestamps, host, process, message)
- Navigating rotated and compressed log archives
- Using simple tools (
less,grep,zless,zgrep,last,lastb,lastlog) to extract the information you need quickly.