Systemd logging

Understanding systemd’s Logging Architecture

systemd provides its own logging component called journald (systemd-journald). Unlike traditional logging that writes plain text files in /var/log, journald collects, structures, and indexes log messages from:

System services started by systemd
The kernel
The initramfs (early boot)
User sessions
Traditional syslog daemons (if present)

Key points specific to systemd logging:

Logs are stored in a binary journal format, not plain text.
Every log entry is a set of key–value fields (structured logging), not just a text line.
Logs can be queried with powerful filters using journalctl.
Journals can be persistent (on disk) or volatile (in memory).

systemd-journald itself is a systemd service, typically defined in systemd-journald.service and systemd-journald.socket.

Journal Storage: Volatile vs Persistent

systemd’s journal can be stored:

Volatile: Only in memory (/run/log/journal), lost on reboot.
Persistent: On disk (/var/log/journal), survives reboots.
Forwarded: Optionally to a syslog daemon or logging backend.

By default on many systems:

If /var/log/journal exists: journald uses persistent storage.
If it does not exist: journald uses volatile storage only.

To enable persistent journaling:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald

After this, journalctl will show logs from previous boots as well, not just the current one.

Using `journalctl` to View Logs

journalctl is the command-line tool for querying the systemd journal. It understands the structured nature of entries and offers many filtering options.

Basic usage

View all logs (can be huge):

  journalctl

Follow logs in real time (similar to tail -f):

  journalctl -f

Show logs only from the current boot:

  journalctl -b

Show logs from the previous boot:

  journalctl -b -1

The -b option accepts boot IDs or offsets (e.g. -b -2 for two boots ago).

Time-based filters

You can restrict entries by time:

Since a given time:

  journalctl --since "2025-01-01 00:00:00"
  journalctl --since "2 hours ago"

Between two times:

  journalctl --since "2025-01-01" --until "2025-01-02 03:00"

Limiting output

To avoid being overwhelmed:

Show only the last N lines:

  journalctl -n 100

Combine with -f to show last 50 lines and follow:

  journalctl -n 50 -f

Show logs in reverse chronological order (newest first):

  journalctl -r

Formatting output

By default, journalctl prints human-readable lines. You can change the format:

Short format (default):

  journalctl -o short

Include syslog-style priority:

  journalctl -o short-precise

JSON output (useful for scripts and tools):

  journalctl -o json
  journalctl -o json-pretty
  journalctl -o json-sse

Show all fields of each entry (very verbose):

  journalctl -o verbose -n 1

Filtering by Units, Services, and Processes

Because journald is integrated with systemd, it knows which unit each log line comes from. This allows very precise queries.

By systemd unit

View logs for a specific service:

  journalctl -u ssh.service
  journalctl -u nginx.service

Follow logs for a service:

  journalctl -u ssh.service -f

Only current boot for a unit:

  journalctl -u ssh.service -b

You can use systemctl to see unit names:

systemctl list-units --type=service

By process ID (PID)

Logs from a specific PID:

  journalctl _PID=1234

This is helpful when debugging specific processes that might be short-lived.

By executable or path

Logs from a specific binary:

  journalctl /usr/bin/sshd

systemd records the path to the executable so this works even across restarts of that service.

Filtering by Fields and Priorities

Each journal entry contains built-in fields (like _PID, _UID, _SYSTEMD_UNIT, PRIORITY, etc.) and user-defined fields from applications.

Priority levels

Systemd uses syslog-style priorities (0 = most severe):

0 – emerg
1 – alert
2 – crit
3 – err
4 – warning
5 – notice
6 – info
7 – debug

View only warnings and more severe:

journalctl -p warning

Or as a range:

journalctl -p err..alert

Field-based filters

You can specify fields directly on the command line:

By user ID:

  journalctl _UID=1000

By systemd unit + priority:

  journalctl -u ssh.service -p warning

By kernel messages only:

  journalctl -k

Fields can be combined; journalctl treats multiple field matches on one command line as a logical OR, and separate groups as logical ANDs. For complex queries, shell quoting becomes important.

Example: logs from sshd or nginx since yesterday:

journalctl -u ssh.service -u nginx.service --since "yesterday"

Inspecting Boot and System State with `journalctl`

systemd’s journal is especially useful for investigating boot problems and service failures.

Boot queries

List boots known to the journal:

  journalctl --list-boots

This shows for each boot:

An index (0 = current, -1 = previous, etc.)
A boot ID (a long hex string)
Start and end timestamps

You can then query with:

journalctl -b -1              # previous boot
journalctl -b 4d3fd8...       # by boot ID

Investigating service failures

When a service fails, use:

systemctl status nginx.service

This shows recent journal entries for that unit. To see a more complete log:

journalctl -u nginx.service --since "1 hour ago"

If a service fails repeatedly, consider:

Using -p warning or higher to focus on errors.
Combining with -b to see only messages from current boot.

Configuring `systemd-journald` (`journald.conf`)

systemd-journald is primarily configured via /etc/systemd/journald.conf. Settings in /usr/lib/systemd/journald.conf (or distribution equivalent) are defaults; /etc/systemd/journald.conf overrides them.

After changes, reload the service:

sudo systemctl restart systemd-journald

Key options you’re likely to tune:

Storage settings

In [Journal] section:

Storage=auto|persistent|volatile|none

auto: Use /var/log/journal if it exists, else /run/log/journal.
persistent: Always use /var/log/journal, create if needed.
volatile: Only keep logs in memory.
none: Don’t store logs; only forward them (if configured).

Example to enforce persistent storage:

[Journal]
Storage=persistent

Size and rotation

These control how much disk space the journal can use. Common options:

SystemMaxUse= – Total disk space limit for system journals.
SystemKeepFree= – How much space to leave free on the filesystem.
SystemMaxFileSize= – Max size per individual journal file.
MaxRetentionSec= – Maximum age of journal entries (e.g. 1month, 7day).

Example:

[Journal]
SystemMaxUse=1G
SystemKeepFree=500M
MaxRetentionSec=1month

Similar options exist for runtime (volatile) logs (RuntimeMaxUse, etc.).

Rate limiting

To prevent log storms from overwhelming the system:

RateLimitIntervalSec=
RateLimitBurst=

Example:

[Journal]
RateLimitIntervalSec=30s
RateLimitBurst=1000

This means: if more than 1000 messages arrive within 30 seconds from the same source, further messages will be dropped until the interval passes.

Forwarding to Syslog and Other Destinations

Even when using journald, you might still run a traditional syslog daemon (like rsyslog or syslog-ng) for compatibility, remote log forwarding, or specialized processing.

In journald.conf:

ForwardToSyslog=yes|no
ForwardToConsole=yes|no
ForwardToKMsg=yes|no
ForwardToWall=yes|no

If ForwardToSyslog=yes and a syslog daemon is running, journald sends a copy of messages to it. The syslog daemon then writes them to /var/log/* or forwards them over the network, following its own configuration.

This architecture enables:

Journald as primary collector and indexer.
Syslog for compatibility and remote forwarding.

Structured Logging with systemd Journal Fields

Every log entry contains fields prefixed by _ (journal-defined) and sometimes without _ (application-defined). Common fields:

_PID – Process ID.
_UID / _GID – User and group IDs.
_COMM – Command name.
_EXE – Path to executable.
_SYSTEMD_UNIT – systemd unit name.
_BOOT_ID – Boot identifier.
_HOSTNAME – Hostname.
MESSAGE – The human-readable message.
PRIORITY – Severity level.

You can inspect all fields of a specific entry:

journalctl -n 1 -o verbose

Or filter with arbitrary fields:

journalctl _COMM=sshd
journalctl _SYSTEMD_UNIT=cron.service _PID=1234

Applications that use the systemd journal APIs (or log via sd_journal_print from libsystemd) can also add their own fields (e.g. REQUEST_ID, USER_NAME), enabling very rich filtering.

Restricting Access and Security Considerations

Journal data often contains sensitive information (usernames, IPs, sometimes even commands and arguments). Access is permission-controlled.

Typical access model:

root can read everything.
Members of special groups (like systemd-journal or adm, depending on distro) can read the journal.
Regular users can usually see:

Their own logs (entries with _UID equal to their UID).
Alternatively, their own session logs.

To allow a user to access full system journals, add them to the appropriate group (example Debian/Ubuntu-like):

sudo usermod -aG systemd-journal alice

After group changes, the user must log out and log back in.

For heightened security:

Limit log retention with MaxRetentionSec=.
Limit disk usage with SystemMaxUse=.
Consider encrypting disks that hold logs if they contain sensitive information.
Be aware that logs can aid both troubleshooting and forensic analysis; set retention according to policy.

Maintenance: Vacuuming and Cleaning Journals

Even with size limits, you may sometimes need to manually remove old journal entries.

journalctl provides vacuuming options:

Remove entries older than a given time:

  sudo journalctl --vacuum-time=14d

Limit total size:

  sudo journalctl --vacuum-size=500M

Limit the number of individual journal files (not usually needed):

  sudo journalctl --vacuum-files=5

These commands will delete the oldest data first until the condition is satisfied.

Logging from Services and User Sessions

Services managed by systemd automatically send their standard output and error to the journal, unless configured otherwise.

In a unit file (e.g. /etc/systemd/system/myapp.service), the StandardOutput and StandardError directives control where logs go:

[Service]
ExecStart=/usr/local/bin/myapp
StandardOutput=journal
StandardError=journal

Common values:

journal – Send to journald (default).
syslog – Forward to syslog (if enabled).
null – Discard output.
tty – Specific terminal.

User sessions also have their own journals. You can inspect them with:

journalctl --user                           # as that user
journalctl --user -u my-user-service.service

User services are managed by systemd --user and store logs in a per-user journal.

Practical Troubleshooting Patterns with systemd Logging

Some useful workflows relying on systemd logging:

After a crash or reboot:
Identify the crash cause:

  journalctl -b -1 -p err..alert

Service flapping (starting and stopping repeatedly):

  journalctl -u myservice.service -b -p warning

Network issues (example for NetworkManager):

  journalctl -u NetworkManager.service --since "30 min ago"

Kernel-related problems:

  journalctl -k -b

Combining systemctl status with targeted journalctl queries provides a tight feedback loop for diagnosing system issues.

Comments

Please login to add a comment.

Don't have an account? Register now!