Table of Contents
Introduction
Logging and auditing form the eyes and ears of a Linux system. They record what has happened, when it happened, and often why. This chapter introduces the overall ideas that connect detailed topics such as systemd logging, traditional logfile structures, audit frameworks, rotation, and custom logging. Later sections in this part of the course will cover each of those areas in depth. Here we focus on the overall purpose, the typical data that gets recorded, and how these mechanisms fit together on a modern Linux system.
Why Logging Matters
Logs are the primary historical record of system activity. They are central for troubleshooting, performance analysis, capacity planning, security investigations, and compliance reporting. Without logs, administrators must guess at what happened. With logs, they can reconstruct events and correlate symptoms to root causes.
Most Linux systems collect logs from several sources. The kernel records low level events related to hardware, drivers, and core OS functions. System services record their own activity and errors. Applications implemented by users and third parties add their own messages. Networking components, firewalls, and authentication layers also contribute information. A complete picture of system behavior emerges only when these sources are viewed together.
Logs are a primary source of truth for diagnosing problems and investigating security incidents. If logging is missing, disabled, or tampered with, reliable reconstruction of events may be impossible.
What Gets Logged
Although each component has its own format and vocabulary, most logs fall into a few broad categories. Operational logs record routine activity such as service start and stop, configuration reloads, scheduled jobs, and normal connections. Error logs describe failures, unexpected conditions, or degraded performance. Security related logs track authentication attempts, authorization decisions, and policy enforcement. Audit logs focus on who did what, often at a finer level than generic security logs.
Typical fields recur across many logging systems. A timestamp records when an event occurred. A facility or component name indicates which program or subsystem produced the entry. A severity or priority marks how important the event is, for example informational versus critical. The message body holds the human readable description, sometimes augmented by structured key value data. Understanding these fields at a general level prepares you to recognize them in systemd logs, classical syslog files, and other logging tools.
Sources of Logs on Linux
On a modern Linux distribution, logs originate from several layers. At the core, the kernel emits messages about devices, memory, filesystems, and other critical subsystems. These are captured by logging daemons or the init system, depending on configuration. System services, such as web servers, database daemons, and background schedulers, usually send their output to a central logger or directly to log files under /var/log. User level applications can write their own logs either through a standard logging interface or via simple file output.
Some services, such as SSH, PAM based authentication modules, and firewalls, are particularly important from a security perspective. They help track login attempts, privilege changes, configuration changes, and blocked network traffic. Other services that might seem less critical, such as printing or cron jobs, can still play a role when diagnosing complex problems, because they reveal what was running at the time of an incident.
Centralization and Structure of Logs
Although many different parts of the system produce log messages, Linux typically centralizes them using a logging framework. Historically this has been a syslog daemon, while on systemd based systems, a central journal service collects messages. Traditional syslog implementations store logs in plain text files, each dedicated to particular facilities or types of events. The init system or logging daemon typically routes messages to these files based on configured rules.
Systemd based systems often maintain a binary journal as the central store and can forward messages into legacy text files for compatibility. This results in two partly overlapping views of the same events, one structured and queryable through specialized tools, and another stored as classic logs under /var/log. Later chapters in this part will explain how to interact with both the journal and traditional logs, but it is important to recognize that they complement each other rather than fully replace one another.
Log Lifecycle and Retention
Logs are not just written once and forgotten. They have a lifecycle that includes collection, storage, rotation, archival, and deletion. Because logs can grow quickly in busy environments, they must be managed carefully to avoid filling disks or slowing down access. Most Linux systems use a rotation mechanism to periodically compress or move old logs and start fresh files. Storage policies determine how long logs are kept locally, when they are archived elsewhere, and when they are finally removed.
Retention requirements depend on both technical and regulatory needs. Developers might need a few days of logs for debugging. Security teams might require months or years of history for forensics and compliance. This leads to layered strategies where recent logs are readily accessible on the system and older data is moved to centralized log servers or long term storage.
Define clear retention and rotation policies. Too little retention undermines investigations, while uncontrolled growth can cause outages when disks fill unexpectedly.
Auditing versus Logging
Logging and auditing are closely related but serve different primary purposes. General logging focuses on operational visibility and troubleshooting. It aims to answer questions about how the system behaves under normal and abnormal conditions. Auditing, on the other hand, explicitly focuses on tracking security relevant actions and policy compliance. It emphasizes who performed an action, what exactly they did, when they did it, and whether the action was authorized.
On Linux, auditing can go beyond ordinary log entries. Specialized frameworks can monitor system calls, file accesses, privilege escalations, and configuration changes. These frameworks are designed to produce a reliable record that holds up to scrutiny, sometimes for legal or regulatory purposes. While general logs might lose entries due to resource limits or misconfiguration, a correctly configured audit system prioritizes completeness and tamper resistance.
The Role of Timestamps and Time Synchronization
Because logs and audits come from many sources, accurate timestamps are critical. Administrators frequently need to correlate events across logs from multiple services or systems. If system clocks are not synchronized, this correlation becomes difficult or misleading. Most production systems therefore use time synchronization services such as NTP or chrony. Although the details of these tools are covered elsewhere, their impact on logging is fundamental.
When analyzing logs, you will often see timestamps in either local time or UTC. Consistency is more important than the particular choice, especially in environments with multiple systems and time zones. Some organizations standardize on UTC in all logs to reduce confusion. Regardless of the convention, maintaining a stable, synchronized time source is a prerequisite for reliable log analysis.
Security and Integrity of Logs
Logs and audit records are themselves security sensitive. Attackers frequently attempt to erase or alter logs to hide their traces. This means that simply having logs is not enough. Their integrity must be protected as well. Several strategies help with this, such as restricting write access to log files, forwarding logs to remote systems, and using append only storage or cryptographic integrity checks.
Access control is equally important. Logs can contain sensitive information, including usernames, partial credentials, IP addresses, and internal hostnames. Permissions on log directories, configuration of logging daemons, and proper use of groups determine who can read various logs. Careless exposure can leak information that simplifies attacks or violates privacy policies.
Protect logs from unauthorized modification and excessive exposure. Integrity of logs is essential for trust, and confidentiality is vital because logs often contain sensitive data.
Working with Logs in Practice
Effective use of logging and auditing involves more than knowing where files are stored. In daily work, administrators search for patterns, extract time ranges, and correlate events across multiple sources. This often involves using standard text processing tools to filter and analyze plain text logs, as well as specialized logging commands for structured systems.
Patterns emerge with experience. Frequent error messages point to chronic misconfigurations. Sudden bursts of authentication failures suggest scanning or brute force attacks. Gaps in logging around critical times indicate potential tampering or misconfiguration. Familiarity with the normal baseline of log output helps highlight anomalies that require investigation.
As systems grow in scale, manual inspection becomes insufficient. Centralized log aggregation, indexing, and search tools become essential. These systems collect logs from multiple hosts, normalize formats, and provide search and alerting capabilities. While detailed configuration of such platforms lies beyond this introductory chapter, logging and auditing on each host remain the foundation.
Relationship to Subsequent Chapters
The remainder of this section will develop the ideas introduced here into concrete skills. You will learn how the systemd journal captures and presents logs. Traditional text logs in /var/log will be explored in detail, with emphasis on their structure and typical usage. Audit specific tools will be introduced to show how Linux can record fine grained security events. Log rotation and retention will be discussed as formal mechanisms rather than general concepts. Finally, you will learn how to add your own logging to scripts and applications so that they integrate smoothly with the rest of the system.
Understanding the common goals behind logging and auditing makes it easier to appreciate each specific tool. Think of this chapter as a conceptual map. The following chapters will fill in the technical details and commands that you will use to make that map operational on real systems.