4.6.1 Systemd logging

Introduction

Systemd logging is built around the journald component, which collects, stores, and indexes log messages from the kernel, user space services, and applications. Instead of each service writing directly to a plain text file in /var/log, systemd prefers that programs write to standard output or use its native logging interface. systemd-journald then captures these messages and keeps them in a structured binary log called the journal.

This chapter focuses on how systemd logging works, how to read and filter logs with journalctl, and how configuration affects what is stored and for how long. Traditional logging in /var/log is discussed elsewhere, so here the emphasis stays on the journal and its tools.

How the journal works

systemd-journald runs as a system service and listens to several sources at once. It receives kernel messages, messages from systemd units, standard output and error streams of services, and messages that programs send via syslog compatibility interfaces. Every message is stored as a set of key value fields, not just a line of text. Typical fields include MESSAGE, _PID, _UID, _SYSTEMD_UNIT, _COMM, and many others that describe who created the log and in what context.

Because entries are structured, search and filtering can be more precise than with plain text. The journal adds metadata automatically. For example, when a systemd service writes to stdout, the journal adds the associated unit name as _SYSTEMD_UNIT, and the executable as _EXE or _COMM. The journal also records timestamps with microsecond resolution and uses monotonic and real time clocks so you can correctly order events across reboots.

By default, the journal is kept in a binary format. This format is not meant to be edited by hand, but you can always extract human readable output through journalctl. Binary storage allows indexing of fields, faster searches, and verification of integrity using checksums, which reduces the chance of unnoticed corruption.

Storage locations and persistence

Systemd logging can be persistent across reboots or limited to the current boot only. The journal stores its files in two possible locations: /run/log/journal and /var/log/journal.

If only /run/log/journal exists, the journal is volatile. This directory is on a temporary filesystem, so logs vanish when the system restarts. If /var/log/journal exists and journald is configured to use it, logs persist across reboots until they are rotated away or vacuumed.

The decision about persistence is influenced by the configuration in /etc/systemd/journald.conf. The Storage option controls whether logs go to volatile memory, persistent disk, or both in certain compatibility modes. It can take values such as volatile, persistent, auto, or none. With auto, journald uses /var/log/journal if present, or falls back to /run/log/journal if not.

Another important aspect is how much space the journal may occupy. Journald tracks size limits such as SystemMaxUse, SystemKeepFree, and SystemMaxFileSize. These control the total size of the system journal, how much free disk space to leave, and how big each individual journal file can be. When limits are reached, older journal files are removed automatically.

Using `journalctl` basics

journalctl is the main tool to view and query the journal. Without options, journalctl shows all available entries from the oldest to the newest, which can be overwhelming on a machine with persistent logs.

To work effectively, you usually start by limiting output to the current boot, the most recent messages, or a specific unit. To show only logs from the current boot, use:

journalctl -b

To view logs in real time similar to tail -f, use the -f option:

journalctl -f

You can combine -b and -f to follow only the current boot’s messages:

journalctl -b -f

By default, journalctl uses a pager, typically less, to display output. You can disable the pager with --no-pager if you want raw output, for example when redirecting logs to a file or script.

To control the amount of output, -n prints only the last N lines. For example:

journalctl -n 50

shows the last 50 log lines, across all boots if persistent logging is enabled.

Filtering by units and services

One of the strengths of systemd logging is that logs are tightly integrated with systemd units. Every service managed by systemd has its output automatically captured and tagged, so you can filter by unit name.

To see logs for a particular service unit, such as sshd.service, use:

journalctl -u sshd.service

This shows logs from that unit over all available boots. To restrict to the current boot only, combine with -b:

journalctl -u sshd.service -b

You can follow logs for a single unit in real time, which is very useful while debugging a service:

journalctl -u sshd.service -f

If a service has multiple units, such as a socket and a service, you can repeat -u to show both:

journalctl -u sshd.service -u sshd.socket

Filtering by unit avoids manual searching in generic system logs and typically gives a focused view of a single component’s behavior.

Time based queries

The journal records precise timestamps for all entries, and journalctl lets you search by time ranges. There are simple options for common periods and more flexible ones for exact times.

To see logs from the current boot, use -b as shown earlier. To see logs from the previous boot, you can specify -b -1. Similarly, -b -2 refers to the boot before that, and so on:

journalctl -b -1

For arbitrary time ranges, --since and --until accept date and time strings. Examples include absolute dates like "2025-01-08 10:00:00" or relative times like "1 hour ago".

For instance, to see logs since yesterday:

journalctl --since=yesterday

To restrict logs to a specific interval, such as from 10:00 to 11:00 on a given date:

journalctl --since="2025-01-08 10:00" --until="2025-01-08 11:00"

These time based filters can be combined with other filters, such as units, to narrow down events related to a problem at a particular time.

Filtering by fields and priorities

Since each journal entry stores multiple fields, you can filter by many criteria in addition to units and time. journalctl supports expressions of the form FIELD=VALUE. Commonly used fields include:

_PID for the process ID.
_UID for the user ID.
_COMM for the command name.
_SYSTEMD_UNIT for the unit name.

For example, to show all messages from a process with PID 1234:

journalctl _PID=1234

To show messages from processes running as a specific user, for instance UID 1000:

journalctl _UID=1000

To see logs generated by a particular command name:

journalctl _COMM=sshd

You can combine field filters by listing them. In that case, they are treated as a logical AND, so only entries matching all conditions are shown.

Log messages also have a priority or severity. Systemd understands standard syslog style priorities: emerg, alert, crit, err, warning, notice, info, and debug. These are ordered from most severe to least. With journalctl -p, you can filter by priority. For example, to show all errors and more severe messages:

journalctl -p err

You can also specify a range, such as -p warning..crit to show warnings through critical messages.

Output formats and exporting logs

By default, journalctl prints logs in a human readable format that resembles a classic syslog line, with a timestamp and message. However, the tool supports several output formats using the -o option.

For script friendly output, -o short and -o short-iso provide variations of compact time stamps. -o verbose prints all fields for each entry. To inspect the full metadata associated with a log message, -o verbose is particularly useful.

Structured formats are important when you want to post process logs with other tools. Two common choices are json and json-pretty. For example:

journalctl -o json-pretty

prints each entry as JSON, with indentation, which you can then parse or ship to a log analysis system.

You can also export the journal in its native binary form for transfer or archival. The --vacuum operations manage size, while journalctl --export can dump the journal to a single binary stream that you can store or send to another machine. On the receiving machine, journalctl --file can read from such an exported file, for example:

journalctl --file=journal-export.bin

If you simply want to save a text copy of certain logs, redirect output to a file:

journalctl -u sshd.service -b --since="today" > sshd-today.log

The resulting file is plain text, suitable for sharing with others or attaching to bug reports.

Configuring `systemd-journald`

The behavior of the journal is controlled by /etc/systemd/journald.conf and snippets under /etc/systemd/journald.conf.d/. Changes in these files usually require reloading or restarting the systemd-journald service to take effect.

Key options include Storage, which decides whether logs are persistent or volatile, and Compress, which enables or disables compression of old journal files. Compression reduces disk usage at a small CPU cost.

Size related options are especially important on production systems. SystemMaxUse sets the maximum disk space that the system journal may occupy in /var/log/journal. SystemKeepFree tells journald to keep at least that amount of disk space free for other uses. SystemMaxFileSize sets a maximum size for individual journal files before rotation. There are corresponding RuntimeMaxUse and RuntimeMaxFileSize settings for the volatile journal under /run/log/journal.

Other options include ForwardToSyslog, ForwardToKMsg, ForwardToConsole, and ForwardToWall. These control whether the journal forwards messages to traditional syslog daemons, the kernel message buffer, local consoles, or wall messages. On systems that still rely on a separate syslog daemon, forwarding is often enabled to keep /var/log files populated.

After modifying the configuration, you can apply changes by running:

systemctl restart systemd-journald

Be cautious with aggressive size limits, since they may cause useful logs to be removed too soon. On the other hand, leaving limits too high on small disks can lead to storage exhaustion.

Rotating and cleaning the journal

The journal manages its own rotation automatically based on file size and configuration limits. Sometimes you may want to clean up older logs manually, especially on systems that have accumulated large journals.

journalctl provides --vacuum-size, --vacuum-time, and --vacuum-files to remove older journal files until certain conditions are met. Each of these options deletes entire journal files, not individual entries.

For example, to limit the total space used by journal files to 1 gigabyte, you can run:

journalctl --vacuum-size=1G

To remove journal files older than 30 days:

journalctl --vacuum-time=30d

Or to keep only the last 5 journal files per journal namespace:

journalctl --vacuum-files=5

These commands apply only to existing files and do not change future automatic limits. To adjust long term behavior, you should also configure size related options in journald.conf.

Working with user level journals

Systemd maintains not only a system wide journal, but also per user journals on some systems. These user journals contain logs for services that run under a particular user’s systemd user instance.

A regular user can access their own logs with journalctl --user. This view includes messages from user services such as graphical applications managed through systemd user units. For example:

journalctl --user

You can combine this with -u to focus on a particular user unit:

journalctl --user -u my-app.service

User logs follow similar rules regarding filtering and formatting, but access is restricted by permissions. Ordinary users cannot see system logs by default, and system administrators cannot see private user logs unless they have sufficient privileges.

On some distributions, user level logging may be disabled or limited. In that case, journalctl --user will either show little or nothing, or will require additional configuration to be useful.

Access control and security considerations

The journal may contain sensitive data, such as error messages with file paths, parts of configuration, or even application payloads if programs log input data. For this reason, access to the system journal is restricted.

Only the root user and members of certain groups, usually systemd-journal or distribution specific log groups, can read all logs. Regular users can typically read only their own logs if user journals are enabled. The permissions on the journal directories and files enforce this.

When debugging or sharing logs with others, you often need to remove or obscure sensitive information. Since journals are binary, it is usually better to export a filtered, text based subset with journalctl that contains only the relevant lines, and then review and sanitize that file before sending it.

Because the journal can be configured to forward logs to other systems, ensure that forwarding and remote collection are done over secure channels and that access controls on remote log servers are properly configured. Although remote logging and centralization are beyond the scope of this chapter, they build upon the same systemd logging concepts presented here.

Summary

Systemd logging, centered around systemd-journald and journalctl, replaces many traditional log file practices with a structured, unified journal. Logs from services, the kernel, and applications are collected in a binary store that supports efficient filtering by units, time, priority, and metadata fields.

You control whether logs persist across reboots and how much space they use through journald.conf and size limits. With journalctl, you can follow logs live, inspect historical entries, export them in various formats, and clean up old journal files when necessary. Understanding these tools provides a solid foundation for working with logs on modern Linux systems that use systemd as their init and service manager.