Table of Contents
Understanding Log Files in `/var/log`
Linux stores most of its system and service logs in the /var/log directory. For system monitoring, this directory is one of the main places you will inspect to understand what is happening on a machine, diagnose problems, or check recent activity.
This chapter focuses on what is unique about /var/log, how log files are typically structured, and how you can read and manage them effectively.
The Role of `/var/log` in System Monitoring
The /var hierarchy is used for variable data that changes during normal system operation. Inside it, /var/log is specifically reserved for log data. Programs and services write status messages, warnings, and errors into files here.
Log files in /var/log are usually plain text. This lets you inspect them with familiar tools such as less, cat, tail, head, grep, and awk. Many command line monitoring workflows are built around reading these logs in real time or searching through them for relevant messages.
Key idea: /var/log is the central on disk location for traditional system and application logs that you can inspect directly as text files.
Typical Structure of `/var/log`
If you list the contents of /var/log, you will usually see a mix of individual log files and subdirectories. The exact set depends on your distribution and which services are installed, but some patterns are common.
There are general system wide logs, which record events from many parts of the system. There are service specific logs stored either directly under /var/log or in subdirectories, for example /var/log/apache2 for a web server, or /var/log/mysql for a database server.
On some modern systems that use systemd, many messages are stored in a binary journal that you access with journalctl. Even on such systems, you still often have traditional log files in /var/log, especially for services that have not fully integrated with the journal.
Common System Log Files
Several log files in /var/log are especially important for monitoring and troubleshooting. While exact names can vary by distribution, the following are commonly seen.
On many systems you will find a main system log file such as /var/log/syslog or /var/log/messages. This file usually contains general information, warnings, and errors from many parts of the system. It is a first stop when something seems wrong, because it collects broad system activity in one place.
Authentication related events are usually logged in files such as /var/log/auth.log or /var/log/secure. These records include login attempts, sudo usage, and other security relevant events. For monitoring security, failed logins or repeated attempts from the same IP address are often visible here.
Kernel specific messages, such as hardware detection or kernel warnings, can appear in /var/log/kern.log or sometimes are mixed into the main system log. When diagnosing driver issues or hardware problems, kernel logs are very useful.
Boot related information can appear in dedicated files such as /var/log/boot.log or in the general system logs. If the system has trouble starting a service at boot, messages in these files may explain why.
System resource usage, especially from processes that are started by cron, might be logged in /var/log/cron or included in a general system log. This can help you verify that scheduled jobs are running and to check whether they are producing errors.
Service Specific Log Directories
Many services create their own log directories inside /var/log. The exact names depend on the software and distribution conventions, but the pattern is consistent.
For example, web servers often log to subdirectories like /var/log/apache2 or /var/log/httpd. In these directories you typically find separate access and error logs, which are very important when monitoring web traffic or debugging failed HTTP requests.
Database servers such as MySQL or PostgreSQL usually have logging locations such as /var/log/mysql or /var/log/postgresql. These logs can show connection attempts, query errors, and other database specific events.
Mail servers, job schedulers, VPNs, and other network services also tend to keep logs under their own subdirectories. When monitoring a particular service, check whether it has its own log directory under /var/log and learn the names of its main log files.
Log File Formats and Timestamps
Most logs in /var/log use a simple line based text format. Each line usually contains a timestamp, the name of the program or service that generated the message, and the message text itself.
You often see timestamps in a human readable form. For example, a line in a classic log file might begin with something like:
Jan 7 13:45:12 hostname program_name[1234]: Informational message hereThe exact format can vary between services. Some use more structured formats, such as key value pairs or JSON like lines. For basic monitoring, recognize where the timestamp and program name appear so that you can quickly see when something happened and which process wrote the message.
When using grep or other tools to filter logs, you often combine searches on the program name or severity keyword and the date or time. Understanding the structure of each log file makes those searches more effective.
Viewing Logs in Real Time
One of the most useful monitoring techniques is watching a log file as messages are added to it. For traditional text logs in /var/log, you can use commands that show new lines as they appear.
A common command is:
tail -f /var/log/syslog
This displays the end of the file and then continues to print new lines. For some systems you may replace /var/log/syslog with /var/log/messages or another active log file. This is helpful when you reproduce a problem and want to see what the system logs at the exact moment it happens.
If you want to follow several logs at once, you can use tools that combine streams, or you can open multiple terminal windows, each using tail -f on a different file. In more advanced setups you might use specialized log viewers or aggregators, but the basic idea of following text logs in real time starts with /var/log.
Log Rotation in `/var/log`
Log files grow over time. Without any control, they could consume a large amount of disk space, which is risky for a production system. To prevent this, Linux systems use log rotation, usually configured through tools like logrotate.
Log rotation periodically moves current log files to new names and starts fresh empty ones. For example, a file such as /var/log/syslog might be rotated to /var/log/syslog.1, and older compressed versions might appear as /var/log/syslog.2.gz, /var/log/syslog.3.gz, and so on.
Important rule: When you are investigating an issue that occurred in the past, always check both the current log file and rotated log files such as file.1, file.2.gz inside /var/log.
You can read compressed rotated logs by using tools that handle compression transparently, or by decompressing them first. The presence of multiple generations of rotated logs allows you to look back in time over a longer period.
For monitoring, rotation has two important effects. First, you must know that older events might be in archived files instead of the current one. Second, when you use tail -f and a log is rotated, some tools can continue following the renamed file, while others may need to be restarted. Understanding that rotation happens in /var/log helps you interpret gaps or apparent restarts in log content.
Permissions and Access to `/var/log`
Since logs often contain sensitive information, such as usernames, IP addresses, and sometimes error messages that include file paths or partial data, file permissions in /var/log are carefully set.
Many log files are readable only by the root user or specific service accounts. When you work as an administrator, you will often need elevated privileges to read certain logs, especially security related ones. For example, an unprivileged user may not be allowed to read authentication logs, while root can read everything in /var/log.
When monitoring a system, always respect these permissions and consider privacy and security implications. Logs can be invaluable for understanding system behavior, but they must also be protected because they can reveal details about the system and its users.
Using `/var/log` in Troubleshooting Workflows
In practical monitoring and troubleshooting, /var/log is part of a simple but powerful workflow. When a problem or alert appears, start by identifying which component is involved. Then look for that component’s logs in /var/log, either in a general system log or in a service specific file.
You then search around the time the problem occurred, often by combining time information with grep filters on keywords such as "error", "fail", or the name of the feature that is misbehaving. You may use tail -f while reproducing the issue, to watch new log lines as the failure happens.
The habit of going directly to the right file in /var/log speeds up diagnosis considerably. Over time, as you learn the typical layout of /var/log on your distribution, you can quickly locate the relevant logs and identify patterns such as recurring errors, warnings that precede failures, or unusual authentication activity.
Relationship Between `/var/log` and Centralized Logging
On larger systems or in networked environments, logs from /var/log are often collected and sent to remote logging servers or aggregators. Even in such setups, applications frequently still write locally to /var/log, and the centralization system reads from there.
For monitoring, this means that local inspection of /var/log gives an immediate, machine specific view, while external tools may present combined data from many machines. Understanding the role of /var/log as the local source of truth helps you verify whether centralized views are complete and up to date.
Even if you rely on graphical dashboards, the plain text files in /var/log remain a critical fallback and reference point when you want direct, low level insight into system behavior.