3.1.4 System logs (journalctl)

Understanding the systemd Journal

Most modern Linux distributions that use systemd also use its logging component: the systemd journal. Unlike traditional text log files in /var/log, the journal stores logs in a structured, binary format and provides a powerful query tool: journalctl.

This chapter focuses on using journalctl to read, filter, and understand system logs. It assumes you already know what systemd is and how to manage services with systemctl.

Basic usage of `journalctl`

The simplest way to view logs from the journal is:

journalctl

This prints the entire journal in chronological order starting from the oldest entry. On most systems this will be a lot of output, so you’ll usually add options to limit what you see.

Common basic options:

-e — jump to the end (most recent entries):

  journalctl -e

-r — reverse order (newest first):

  journalctl -r

-n N — show last N lines:

  journalctl -n 50

-f — follow logs in real time (similar to tail -f):

  journalctl -f

You can combine them:

journalctl -f -u ssh
journalctl -n 100 -r

Time-based filtering

journalctl can show logs from a particular time range. The most useful options are:

--since — starting time
--until — ending time

You can use absolute times:

journalctl --since "2025-01-01 10:00:00" --until "2025-01-01 12:00:00"

Or relative times:

journalctl --since "1 hour ago"
journalctl --since "yesterday"
journalctl --since "2025-01-01"

You can combine time filters with other filters:

journalctl -u nginx --since "today"
journalctl -p err --since "10 minutes ago"

Filtering by unit, service, and more

Because the journal is integrated with systemd, you can easily show logs related to specific units and system components.

By systemd unit

The most common filter is by unit (-u):

journalctl -u ssh
journalctl -u nginx.service
journalctl -u docker.service -f
journalctl -u NetworkManager.service --since "today"

You can specify -u multiple times:

journalctl -u nginx -u php-fpm --since "1 hour ago"

System logs only vs user logs

By default, journalctl shows system-wide logs. You can focus on:

System-level logs:

  journalctl --system

Logs for the current user only:

  journalctl --user

User unit logs (for user services) are often inspected with:

journalctl --user -u my-user-service --since "today"

Boot-specific logs

The journal records each boot separately. To see logs from the current boot only:

journalctl -b

To see logs from previous boots, add an offset:

-b -1 — previous boot
-b -2 — two boots ago, etc.

Examples:

journalctl -b -1
journalctl -b -1 -u nginx
journalctl -b -1 --since "10:00" --until "11:00"

To list available boots with their IDs:

journalctl --list-boots

This prints a table with:

offset (e.g. -1, 0, 1, …)
boot ID (a long hex string)
start and end time of the boot

You can also use a specific boot ID:

journalctl -b 7ed2f1a0b87b4a45acb2fe6c83d7d0a1

Filtering by priority (severity)

Every log message has a priority (also called severity). Common levels, from most severe to least:

0 — emerg (system is unusable)
1 — alert
2 — crit
3 — err
4 — warning
5 — notice
6 — info
7 — debug

journalctl lets you filter by priority with -p. You can use the name or the number:

Logs with priority err (3) and more severe:

  journalctl -p err
  journalctl -p 3

Logs from warning (4) and more severe:

  journalctl -p warning

Combined examples:

journalctl -u nginx -p warning --since "today"
journalctl -b -p err

Searching and matching text

Although the journal is structured, you will often want to search for a text pattern.

Simple text search with `grep`

You can pipe journalctl output into grep:

journalctl -u nginx --since "today" | grep error
journalctl -b | grep -i failed

Use -i with grep for case-insensitive matches.

Structured field matching

Journal entries contain many fields (like SYSLOG_IDENTIFIER, _PID, _UID, etc.). You can filter directly on fields:

Basic examples:

By executable name (syslog identifier):

  journalctl SYSLOG_IDENTIFIER=cron

By process ID:

  journalctl _PID=1234

By user ID:

  journalctl _UID=1000

By kernel messages only:

  journalctl -k

You can see all fields of a particular entry using verbose output (see next section).

Output formatting and detail level

Standard vs verbose output

By default, journalctl prints a human-friendly format with date, host, unit, and message.

To see all fields for each log entry, use verbose mode:

journalctl -o verbose -n 5

This shows every metadata field the journal stored, such as _PID, _UID, _SYSTEMD_UNIT, _BOOT_ID, and more.

Output formats (`-o`)

journalctl supports different output modes via -o. Common ones:

-o short — default concise format
-o short-iso — ISO 8601 timestamps:

  journalctl -o short-iso -n 20

-o json — one JSON object per line:

  journalctl -o json -n 5

-o json-pretty — pretty-printed JSON:

  journalctl -o json-pretty -n 5

-o cat — message text only (no timestamps or metadata):

  journalctl -u nginx -o cat

The JSON formats are useful when programmatically analyzing logs with scripts or other tools.

Working with persistent vs volatile logs

Depending on distribution and configuration, journal logs may be:

Volatile: stored in memory only; lost on reboot
Persistent: stored on disk; survive reboots

The journal’s persistent data is usually in /var/log/journal. Some key points:

If /var/log/journal exists, systemd-journald will normally store logs persistently.
If it doesn’t, logs may be kept in /run/log/journal (tmpfs), which is cleared on reboot.

You can check journal disk usage with:

journalctl --disk-usage

This shows how much space current logs use. You may configure limits (covered in a separate chapter on logging configuration and systemd internals), but as an operator you should at least know this command.

Limiting and cleaning logs

Although log retention and limits are mostly configured in journald’s config files, journalctl provides a few maintenance options (use with caution and usually as root).

Deleting old logs by time

To remove entries older than a certain time:

sudo journalctl --vacuum-time=7d

This keeps logs from the last 7 days and deletes older ones. You can also use hours or months:

sudo journalctl --vacuum-time=12h
sudo journalctl --vacuum-time=3months

Limiting by size

You can instruct journalctl to shrink the journal to a target size:

sudo journalctl --vacuum-size=1G

This deletes the oldest entries until the total disk space used by the journal is at or below 1 GB.

Limiting by number of files

You can also limit the number of individual journal files:

sudo journalctl --vacuum-files=10

Note that these are one-off commands; permanent limits belong in journald configuration, not in journalctl.

Kernel messages and boot problems

The journal stores kernel messages, which are often critical for diagnosing hardware or boot issues.

Viewing kernel messages

Use -k to show only kernel logs:

journalctl -k
journalctl -k -b      # kernel logs for current boot
journalctl -k -b -1   # kernel logs for previous boot

Combine with priority filters:

journalctl -k -p warning -b

This is particularly useful for spotting driver issues, hardware failures, or kernel panics.

Checking last boot issues

To inspect what went wrong in the most recent boot:

journalctl -b -1 -p warning
journalctl -b -1 --since "5 minutes before" --until "10 minutes after"

This helps pinpoint errors during startup, such as services failing to start or devices not initializing.

Per-service troubleshooting workflow

When debugging a specific systemd service, combine systemctl and journalctl:

Check the service status:

   systemctl status nginx

This often shows the last few relevant log lines from the journal.

View full logs for that service:

   journalctl -u nginx

Focus on the latest attempt:

   journalctl -u nginx -b

Follow logs in real time while starting/restarting the service:

   journalctl -u nginx -f

Then in another terminal:

   sudo systemctl restart nginx

This realtime view helps you see error messages exactly as they appear during service startup or operation.

Saving and exporting logs

Sometimes you need to save logs for later analysis, sharing, or archival.

Plain text export

To save filtered logs as plain text:

journalctl -u nginx --since "today" > nginx-today.log
journalctl -b -p err > errors-this-boot.log

Binary export and import

The journal’s native binary format can be exported and imported, preserving full metadata:

Export:

  sudo journalctl -b -o export > boot.journal

Import into another system’s journal:

  sudo journalctl --file=boot.journal

Or specify --file when viewing:

journalctl --file=boot.journal -n 50

This can be useful for forwarding logs for offline analysis.

Using `journalctl` with multiple files or chroots

You can point journalctl at arbitrary journal directories or files, which is helpful for:

Inspecting logs from a mounted disk of another system
Looking at logs inside a chroot or container filesystem