4.6.4 Log rotation

Why Log Rotation Matters

Log files grow every time a program writes a message. On a busy system, some logs can reach gigabytes in a few days. If they are never limited or cleaned, they can fill a filesystem, slow down searches in the logs, and make backup and analysis difficult.

Log rotation is the automatic process of periodically closing a log file, renaming or compressing it, and starting a new, empty log file. Old rotated logs are then optionally compressed and removed after a defined time or number of files.

On most Linux systems, this is handled by a dedicated tool that runs from cron or systemd timers and applies rotation rules to many log files at once based on configuration files.

The Role of `logrotate`

The standard tool for log rotation on many distributions is logrotate. It reads configuration files, applies rules to each log file, and performs rotation actions. It does not run continuously. Instead, a scheduler runs it periodically, typically once per day.

A basic logrotate run follows this pattern. It parses main and per-application configuration, tests each log file against its rules, rotates files that meet the conditions, optionally compresses or deletes older ones, and triggers any post-rotation actions such as asking a daemon to reopen its log file.

You will often see a daily cron job such as /etc/cron.daily/logrotate or a systemd timer like logrotate.timer which calls the logrotate binary with the main configuration file.

Configuration Locations and Structure

logrotate reads a main configuration file and then includes additional configuration snippets. The main file is usually /etc/logrotate.conf. Inside it, there are global options and one or more include directives which pull in configuration from a directory such as /etc/logrotate.d.

The structure you are likely to find is:

• The main file defines defaults that apply to all log files unless overridden.
• Each file under /etc/logrotate.d usually belongs to one package or service and defines how that specific service’s logs are rotated.

Configuration is organized in stanzas. A stanza is a block that begins with a log file path or list of paths and curly braces. For example:

/var/log/myapp.log {
    daily
    rotate 7
    compress
}

Everything inside the braces applies to the log files listed before the opening brace.

Time-based vs Size-based Rotation

There are two common triggers for rotation, time-based rules and size-based rules. You can use them separately or sometimes in combination.

Time-based rotation uses directives such as daily, weekly, monthly, or yearly. For instance:

/var/log/example.log {
    weekly
}

This instructs logrotate to rotate that log about once per week, usually on the first run after a full week has elapsed since the last rotation.

Size-based rotation uses the size directive. For example:

/var/log/example.log {
    size 100M
}

This means the log is rotated when its size is equal to or larger than 100 megabytes. Valid suffixes include k, M, or G for kilobytes, megabytes, or gigabytes.

By default, logrotate typically uses a time-based rule. If you specify both time and size, behavior depends on additional options such as notifempty or minsize. If you want rotation to happen whenever both the age and the size conditions are met, you can use minsize to require a minimum size in addition to time.

Keeping and Naming Old Logs

When a log is rotated, the current file is renamed, and a new empty file is created so that the application can continue writing logs. logrotate assigns names to old files and decides how many to keep based on specific directives.

The rotate directive controls how many old logs are preserved. For instance:

/var/log/example.log {
    weekly
    rotate 4
}

This configuration keeps four previous generations. When rotation happens and this limit is exceeded, the oldest log is deleted. If rotate 0 is used, the old log is removed immediately and only the current file exists.

Names of rotated logs follow numbering or date-based patterns. The default pattern is numeric, where files are renamed to /var/log/example.log.1, then .2, .3, and so on, possibly combined with compression suffixes such as .gz.

You can switch to date-based naming using the dateext option. For example:

/var/log/example.log {
    daily
    dateext
}

With dateext, rotated logs might be named like /var/log/example.log-2024-12-31.gz. This makes it easier to see when each file was created and avoids renumbering, at the cost of potentially more scattered names.

Additional options such as dateformat can control the exact format of the date portion. This is useful if you have external tools that expect a particular naming pattern.

Compression and Storage Savings

Rotated logs are often compressed to reduce disk usage. Since log files are plain text, compression typically yields a large reduction in size.

The compress directive in a stanza tells logrotate to compress old logs. Without further options, the compression format is usually gzip, which produces files ending in .gz. To keep the most recent rotated file uncompressed for easy searching, you can use delaycompress together with compress. For example:

/var/log/example.log {
    weekly
    rotate 8
    compress
    delaycompress
}

In this configuration, the first time the file is rotated, the previous .1 file is left uncompressed. On the next rotation, that older file is compressed. This makes it easier to inspect the most recent historical log without decompression.

There are options such as compresscmd, uncompresscmd, and compressext to adjust the compression program and suffix if your distribution supports them. For instance, you could replace gzip with xz or bzip2 at the cost of more CPU time during rotation.

Handling Empty or Missing Logs

In practice, some log files are not always present or might be empty. logrotate has specific directives to control behavior in these cases.

The missingok directive means that if the log file does not exist, logrotate will silently skip it instead of treating it as an error. This is common in package-provided configurations where the log might not exist on every system or at all times.

The notifempty directive instructs logrotate not to rotate an empty log file, even if time or size thresholds are otherwise met. This prevents a series of empty rotated files.

The opposite of notifempty is ifempty, which is often the default. With ifempty, rotation can occur even if the file currently has size zero. You usually choose notifempty when you only care about logs that actually contain content.

Copying vs Moving and Creating New Files

When rotation occurs, you usually want the current log to be closed, moved to a new name, and then a new empty file created. However, some services do not respond well if you simply move or truncate their log files while they keep the file handle open.

To adapt to these behaviors, logrotate offers several modes.

The default behavior is to rename the current file and then create a new one. The create directive controls the permissions, ownership, and group of the new file. For instance:

/var/log/example.log {
    weekly
    create 0640 root adm
}

This means that after rotation, a new file is created with mode 0640, owned by user root and group adm.

For services that cannot handle renames well, you can use copytruncate. This option copies the contents of the current log to the rotated file and then truncates the original file to zero length. The file name and file handle stay the same from the application point of view. For example:

/var/log/example.log {
    size 50M
    copytruncate
    rotate 5
}

copytruncate is convenient but can potentially lose some log messages if the application writes to the file while copying is taking place. This is an inherent race condition between copying and writing. Use it only if you cannot configure the application to handle reopening a new log.

For multi-instance logs or log files that do not need to be recreated, you can use nocreate to avoid making a new logfile. This is less common with typical system services, which usually want a writable log ready after rotation.

Coordinating Rotation with Services

Many daemons keep open file descriptors to their log files and do not automatically detect when files are renamed or replaced. If they continue writing to an old file that logrotate has moved aside, the new empty file will stay unused and logs will go into files you thought were archived.

To avoid that, you can define script hooks that run before or after rotation inside a logrotate stanza. The most common are postrotate and endscript. Together they define a shell script that runs after rotation has finished. For example, for a service managed by systemd:

/var/log/mydaemon.log {
    weekly
    rotate 4
    compress
    postrotate
        systemctl reload mydaemon.service >/dev/null 2>&1 || true
    endscript
}

In this configuration, after rotating the log files, logrotate will send a reload request to the daemon. The daemon should then reopen its log file and start writing to the new one. If the command fails, the final || true ensures that logrotate does not treat that as a fatal error.

You can also use prerotate to run commands before rotation. This is useful if you need to flush caches, synchronize data, or temporarily pause logging. For more complex scripts, you can call an external script file from within the block.

Using the `sharedscripts` and `su` Options

Sometimes a single configuration block lists multiple log files belonging to one service. By default, logrotate executes the postrotate and prerotate scripts once per log file. To run them only once per block, you can use the sharedscripts directive.

For example:

/var/log/myapp/*.log {
    weekly
    rotate 4
    compress
    sharedscripts
    postrotate
        systemctl reload myapp.service >/dev/null 2>&1 || true
    endscript
}

With sharedscripts, the postrotate script runs once even if multiple .log files under /var/log/myapp are rotated.

The su directive lets logrotate drop privileges when creating or handling log files so that it can manage logs for services that do not run as root. It takes a user and group:

/var/log/myapp/*.log {
    daily
    rotate 7
    compress
    su myuser mygroup
    create 0640 myuser mygroup
}

This is particularly important on systems where the cron job or systemd unit runs logrotate as root, but you want rotated logs to be owned by a non-root service user.

Testing and Forcing Rotation

Before you rely on a new configuration, it is useful to test it. logrotate provides flags for dry runs and for forcing rotation.

A dry run simulates what logrotate would do without actually rotating, compressing, or deleting anything. The -d or --debug option produces detailed output. For example:

logrotate -d /etc/logrotate.conf

This command reads the configuration and prints actions that would be taken, but it leaves the actual log files unchanged. This is very useful when you add a new stanza or modify options.

To force rotation even if time or size conditions are not met, you can use the -f or --force option:

logrotate -f /etc/logrotate.conf

This is often used when you want to manually trigger a rotation during testing or after making a change to a service that writes logs.

If you want to test a particular configuration file snippet, you can pass it directly on the command line instead of the main configuration. For example:

logrotate -d /etc/logrotate.d/myapp

This isolates debugging to the rules for a single application without running through all system logs.

Typical Example: Rotating a Custom Application Log

Consider a custom application that writes to /var/log/myapp/app.log. You want to rotate its logs daily, keep two weeks of history, compress older logs, skip empty files, and tell the service to reopen its logs after rotation.

A typical configuration snippet in /etc/logrotate.d/myapp could look like this:

/var/log/myapp/app.log {
    daily
    rotate 14
    compress
    delaycompress
    notifempty
    missingok
    create 0640 myapp myapp
    postrotate
        systemctl reload myapp.service >/dev/null 2>&1 || true
    endscript
}

Here, daily and rotate 14 keep logs for about two weeks. compress and delaycompress save space while leaving the most recent rotated file uncompressed for easier manual inspection. notifempty ensures that only logs with content are rotated. missingok prevents errors if the log file has not yet been created. The create line ensures that new logs have the correct ownership and permissions. Finally, the postrotate block causes the service to reopen its log file after each rotation.

This pattern is easily adapted to other services by changing file paths, user and group, rotation frequency, and retention count.

Managing Rotation Frequency vs Retention

Two related but different choices in log rotation are how often to rotate and how long to keep historical logs. Frequency is controlled by daily, weekly, and similar options. Retention is controlled by rotate N and sometimes by external archiving systems.

If you rotate very frequently but keep many generations, you will have more files but each file will be smaller. If you rotate less often but keep fewer generations, each file will be larger but you will have fewer files in total. There is no universal best practice, since it depends on:

The rate at which logs grow. A very noisy application may need size based or daily rotation to avoid large files, while a quiet one might be fine with weekly rotation.

Your analysis and compliance needs. If you need to keep logs for months or years, you might rotate weekly for local convenience and then move old rotated logs to long term archival storage instead of keeping all locally.

Disk space and backup costs. Compressed rotated logs are usually included in backups, so their total size has an impact on backup time and storage.

Balancing these factors is a design decision for each system or application. logrotate gives you the knobs to tune but does not decide the policy for you.

Interaction with Other Logging Systems

Some modern setups use log forwarding, central logging servers, or journaling systems that already manage rotation or retention. In such environments, you must be careful not to duplicate or conflict with existing mechanisms.

If a logging agent already rotates its own files, you may not need or want a separate logrotate stanza for those logs. In contrast, if you have logs written directly to files under /var/log by older services, they almost always require explicit logrotate rules.

When system logs are primarily stored in a journal and then forwarded elsewhere, log rotation for file based logs may be less central, but file based application logs will still need attention. The main pattern remains the same: use logrotate to control size and retention of local text log files and coordinate with services so they reopen logs correctly after rotation.