Table of Contents
Understanding `/var/log` in Context
On most Linux systems, /var/log is the central place where system and application logs are stored. From a monitoring perspective, it’s where you:
- Investigate problems (boot issues, service crashes, authentication failures).
- Track security-related events.
- Correlate historical events with performance anomalies.
This chapter focuses on what you’ll typically find inside /var/log, how the files are structured and rotated, and how to read them efficiently for monitoring and troubleshooting.
How Logging to `/var/log` Works (High Level)
Modern systems often use systemd-journald plus a traditional syslog daemon (like rsyslog or syslog-ng):
systemd-journaldcollects logs from the kernel, systemd services, and some applications.- A syslog daemon (if present) receives log messages and writes them into text files under
/var/log.
You don’t need to configure this here; just understand that:
- Applications log via syslog or directly to their own files.
- Log files in
/var/logare mostly plain text and viewable withless,tail,grep, etc.
Structure of `/var/log`
Typical top-level contents might look like:
/var/log
├── auth.log # or secure
├── boot.log
├── dmesg
├── kern.log
├── messages
├── syslog
├── faillog
├── journal/ # systemd journal (binary)
├── apt/ # package manager logs (Debian/Ubuntu)
├── dnf.rpm.log # package manager logs (Fedora/RHEL)
├── httpd/ or apache2/ # web server logs
├── nginx/
├── samba/
├── Xorg.0.log
└── ... many more ...The exact set of files and directories varies by distribution and installed software, but there are common patterns:
- System-wide logs:
syslog,messages,kern.log,dmesg, etc. - Security/authentication logs:
auth.logorsecure,faillog, etc. - Service-specific directories:
httpd/,nginx/,mysql/, etc. - Package manager logs:
apt/,dnf.rpm.log,yum.log,zypper.log, etc. - Binary journal:
journal/(used withjournalctl—covered elsewhere in systemd logging).
Key System Log Files
Exact filenames differ by distribution, but these are common categories and where they usually appear.
General System Logs
`/var/log/syslog` (Debian/Ubuntu)
- Contains: General system messages, daemon logs, application messages that use syslog.
- Use it for:
- High-level overview of what the system is doing.
- Tracing service startup and shutdown messages.
- Correlating events across multiple services by timestamp.
Basic usage:
sudo less /var/log/syslog
sudo tail -f /var/log/syslog
On systems without syslog, this file may not exist; instead, journalctl is used.
`/var/log/messages` (RHEL, CentOS, Fedora, some others)
- Similar role to
/var/log/syslogon Debian-based systems. - Contains generic, non-authentication system messages.
Usage:
sudo tail -f /var/log/messages
sudo grep -i error /var/log/messagesKernel-Related Logs
`/var/log/kern.log` (often on Debian-based systems)
- Contains: Kernel messages (hardware detection, driver issues, kernel warnings).
- Use it for:
- Hardware-related problems.
- Disk, controller, or network driver issues.
- Kernel module load/unload logs.
Example:
sudo grep -i "error" /var/log/kern.log
sudo tail -f /var/log/kern.log`/var/log/dmesg`
- Snapshot of kernel ring buffer, usually captured at boot.
- Use it for:
- Boot-time hardware detection.
- Initial driver messages.
- Note that
dmesgcommand reads from the live kernel buffer;/var/log/dmesgis static (until rotated/overwritten).
Usage:
sudo less /var/log/dmesgBoot and Startup Logs
`/var/log/boot.log`
- Contains: Messages from the boot process, particularly from the init system and startup scripts.
- Use it for:
- Slow boot troubleshooting.
- Errors during service startup at boot.
Example:
sudo less /var/log/boot.log
On systemd-based systems, more detailed boot logs often come via journalctl -b, but boot.log still provides a plain-text summary when present.
Authentication and Security Logs
`/var/log/auth.log` vs `/var/log/secure`
- Debian/Ubuntu:
/var/log/auth.log - RHEL/CentOS/Fedora/openSUSE:
/var/log/secure
These files log:
- SSH logins (successful and failed).
sudousage (who ran which commands).- Local login attempts (TTY, console).
- Some PAM-related events.
Monitoring examples:
# Show recent auth events (Debian/Ubuntu)
sudo tail /var/log/auth.log
# Show failed SSH attempts
sudo grep "Failed password" /var/log/auth.log
# On RHEL/Fedora-based systems
sudo tail /var/log/secure
sudo grep "Failed password" /var/log/secure`/var/log/faillog`
- Binary log of failed login attempts.
- Viewed with
faillogtool, not a text editor.
Example:
sudo faillog # Summary of failed logins
sudo faillog -u usernameUseful for security monitoring (brute-force login attempts).
`/var/log/btmp` and `/var/log/wtmp`
/var/log/wtmp:- Records all logins and logouts.
- Viewed with
last. /var/log/btmp:- Records failed login attempts.
- Viewed with
lastb.
Examples:
# Show login history
last # reads /var/log/wtmp
# Show failed login attempts
sudo lastb # reads /var/log/btmpThese are binary logs; do not edit them directly.
Service-Specific Logs in `/var/log`
Many services write their own logs under /var/log or a subdirectory. You’ll use these heavily for monitoring specific services.
Web Servers
Apache HTTPD
Typical locations:
- Debian/Ubuntu:
/var/log/apache2/ - RHEL/CentOS/Fedora:
/var/log/httpd/
Common files:
access.log(oraccess_log):- Every HTTP request (client IP, timestamp, URL, status, user agent).
error.log(orerror_log):- Server errors, configuration issues, PHP errors (when integrated), etc.
Usage examples:
# Follow Apache access log
sudo tail -f /var/log/apache2/access.log
# Check for 500 Internal Server Errors
sudo grep " 500 " /var/log/apache2/access.logNginx
Typical location: /var/log/nginx/
Common files:
access.logerror.log
Monitoring example:
sudo tail -f /var/log/nginx/access.log
sudo grep -i "error" /var/log/nginx/error.logDatabase Servers
Examples (paths may vary):
- MySQL/MariaDB:
/var/log/mysql/error.log/var/log/mysqld.log(RHEL-style)- PostgreSQL:
- Often in
/var/log/postgresql/or under the data directory configured inpostgresql.conf.
Use these to:
- Investigate connection errors.
- Check slow queries (if slow query log is enabled).
- Monitor crashes or restarts.
Other Common Service Logs
- SSH daemon: Messages usually go into
/var/log/auth.logor/var/log/secure, sometimes with additional files like/var/log/sshd.logdepending on syslog config. - Mail servers (Postfix, Exim, etc.):
/var/log/mail.log,/var/log/mail.err,/var/log/maillog.- Samba:
/var/log/samba/, with per-service or per-client logs.- Cron:
- Debian/Ubuntu:
/var/log/cron.logor inside/var/log/syslog. - RHEL/Fedora:
/var/log/cron.
Each service typically documents its log file locations in its configuration files or manual pages.
Package Management Logs
These logs help you answer: What changed on this system and when?
Debian/Ubuntu — `/var/log/apt/`
Key files:
history.log:- High-level record of package installations, upgrades, and removals.
term.log:- Terminal output produced during APT operations (useful for error details).
Examples:
less /var/log/apt/history.log
grep "install" /var/log/apt/history.logRHEL/Fedora — `dnf.rpm.log` and `yum.log`
Common files:
/var/log/dnf.rpm.logor/var/log/dnf.log/var/log/yum.log(older systems)
Check which packages were installed/updated and when:
sudo less /var/log/dnf.rpm.log
sudo grep "install" /var/log/yum.logOther managers
- openSUSE (zypper):
/var/log/zypper.log - dpkg:
/var/log/dpkg.log(Debian/Ubuntu).
These are useful for correlating new issues with recent package changes.
X and Desktop Logs
On desktop systems:
- Xorg server log:
/var/log/Xorg.0.log - Contains info on graphics drivers, resolutions, input devices.
- Some desktop environments or display managers (like
lightdm,gdm,sddm) have their own logs, often under/var/log/or/var/log/<managername>/.
These are primarily used for troubleshooting graphical display problems.
Log Rotation in `/var/log`
Log files grow over time; logrotate (or a similar tool) manages their size, retention, and compression.
Recognizing Rotated Logs
In /var/log, you’ll often see patterns like:
syslog
syslog.1
syslog.2.gz
syslog.3.gzOr:
messages
messages-20241115
messages-20241108.gzMeaning:
syslog: current log file.syslog.1: previous file (uncompressed).syslog.2.gz,syslog.3.gz: older logs, compressed to save space.
To search in compressed logs, either decompress them or use zgrep, zless, etc.:
zgrep "error" /var/log/syslog.2.gz
zless /var/log/messages-20241108.gzBasic `logrotate` Behavior (Conceptual)
While configuration details are separate:
logrotateis usually run daily (via cron or systemd timer).- For each log, it can:
- Rotate based on size or age.
- Compress old logs.
- Keep a specified number of files.
- Run post-rotate scripts (e.g., to reload a service).
Configuration is normally in /etc/logrotate.conf and /etc/logrotate.d/. Understanding that logs rotate helps when you’re looking for older events.
Access Control and Permissions
Many logs in /var/log are readable only by root or certain groups, because they contain sensitive data (usernames, IPs, command histories via sudo, etc.).
Check permissions:
ls -l /var/log/auth.log
ls -ld /var/log
If you’re troubleshooting as a regular user, you may need sudo:
sudo less /var/log/secure
sudo tail -f /var/log/messagesAvoid changing permissions on sensitive logs just to make them readable; that can introduce security risks.
Practical Monitoring Examples Using `/var/log`
Here are some focused tasks tied directly to /var/log for day-to-day monitoring and troubleshooting.
Monitor a Service for Errors
For example, Apache on Debian/Ubuntu:
# Follow the error log for new entries
sudo tail -f /var/log/apache2/error.log
# Look for 'Segmentation fault' in recent history
sudo grep -i "segmentation fault" /var/log/apache2/error.logInvestigate SSH Brute-Force Attempts
Debian/Ubuntu:
sudo grep "Failed password" /var/log/auth.log | tail
sudo grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -nrRHEL/Fedora:
sudo grep "Failed password" /var/log/secureCorrelate System Crash with Package Updates
Suppose a service started failing yesterday:
- Check auth/system logs for failure time:
sudo grep "myservice" /var/log/syslog- Check if any packages were upgraded near that time:
# Debian/Ubuntu
less /var/log/apt/history.log
# RHEL/Fedora
less /var/log/dnf.rpm.logSearch Historic Logs Including Rotated Files
To search all syslog instances (current + rotated, compressed):
sudo zgrep "myservice" /var/log/syslog*
Adapt for messages or other logs as needed.
Good Practices When Working with `/var/log`
- Use
sudoread-only: Preferless,tail,grep—avoid editing log files directly. - Don’t truncate logs casually: Clearing logs can hinder troubleshooting and auditing.
- Know where your service logs: Check service documentation or configuration (e.g.,
ErrorLogandCustomLogin Apache config). - Be aware of rotation: If an event is older than expected, check rotated and compressed files.
- Combine with other tools: Use
/var/logalongside performance tools (liketop,vmstat, etc.) to correlate resource spikes with logged events.
The more familiar you become with the layout and contents of /var/log, the faster you’ll be able to diagnose and monitor your systems in real-world scenarios.