Backup strategies

Why Backup Strategy Matters

A backup strategy is more than “run rsync sometimes.” It is a plan that answers:

• What will be backed up
• Where it will be stored
• How often it will run
• How long it will be kept
• How you will restore it (and how fast you need to)

This chapter focuses on planning and design, not the tools themselves (those are covered in Using rsync, Creating tar archives, and Using snapshot systems).

Core Concepts in Backup Planning

RPO and RTO (How Much Can You Lose?)

Two key ideas shape your backup strategy:

• RPO (Recovery Point Objective)
  The maximum acceptable amount of data you can lose, measured in time.
  Examples:
• RPO = 24 hours → daily backups are enough.
• RPO = 1 hour → you need at least hourly backups or continuous replication.
• RTO (Recovery Time Objective)
  How long you can afford the system to be down before it’s restored.
  Examples:
• RTO = 4 hours → simple file restores might be OK.
• RTO = 5 minutes → you likely need hot standbys, snapshots, or failover systems.

Your RPO and RTO should drive all other choices in this chapter.

What to Back Up (Scope)

Not everything needs the same level of protection.

Typical categories:

• Critical data
• Databases, business data, code repositories, configuration files.
• System configuration
• /etc, service configs, scripts in /usr/local, crontabs, custom units.
• User data
• Home directories, documents, mail, shared data.
• System binaries
• Often not worth backing up in depth on modern distributions, because you can usually reinstall from packages.

Decisions to make:

• Which directories are must have vs nice to have?
• Do you back up whole systems or just data and configs, then rebuild from packages when needed?
• Are some systems stateless (no need for heavy backups) and others stateful (require strong protection)?

---

Types of Backups

Full Backups

A full backup copies all selected data.

• Pros
• Simple to understand.
• Faster restores (everything in one place).
• Good baseline to test your restore process.
• Cons
• Slow to create.
• Uses the most storage.
• Often impractical to run frequently on large systems.

Use full backups:

• Periodically (e.g., weekly/monthly) as a base for other types.
• For small systems where data size is limited.

---

Incremental Backups

An incremental backup stores only data changed since the last backup of any kind.

• Pros
• Very efficient in storage.
• Usually faster to run after the first full backup.
• Cons
• Restores can be slower and more complex:
• Need the last full backup plus all following incrementals.
• If one incremental is corrupted, you may lose the chain after it.

Example schedule:

• Sunday: full
• Mon–Sat: incremental (changes since previous day’s backup)

This is a common approach when data changes daily and you want to save space.

Differential Backups

A differential backup stores data changed since the last full backup.

• Pros
• Faster restores:
• Need the last full backup plus the latest differential.
• More robust than long incremental chains.
• Cons
• Differential backups grow larger as the week passes.
• More storage and time than incrementals, less than repeated full backups.

Example schedule:

• Sunday: full
• Mon–Sat: differential (each day = changes since Sunday)

This is a compromise between restore simplicity and storage usage.

Snapshots vs Traditional Backups

Snapshots are covered in depth in Using snapshot systems, but for strategy:

• Snapshots (e.g., LVM, Btrfs, ZFS)
• Very fast to create.
• Usually efficient for short-term rollback.
• Often stored on the same disk or pool → not a complete backup on their own.
• Traditional backups
• Copy data to a different location (ideally off the system).
• Better protection against disk failure or system loss.

A good strategy often uses both:

• Frequent local snapshots for fast local rollback.
• Periodic external backups for disaster recovery.

---

The 3-2-1 Rule

A common guideline for reliable backups:

1. 3 copies of your data
• 1 primary (live data) + 2 backups
2. 2 different media
• For example: local disk + remote storage, or disk + tape, or disk + cloud.
3. 1 off-site copy
• Stored in another location (or in a different region/cloud).

Why it matters:

• Protects against disk failure, accidental deletion, and local disasters (fire, theft, flooding, ransomware on the main system).
• Even on home systems, having at least one copy not physically next to the original is valuable (e.g., encrypted remote backup).

---

Backup Frequency and Retention

Choosing Backup Frequency

Tie this to RPO:

• Low change rate, tolerant of data loss (e.g., lab server):
• Weekly full or monthly full + daily incremental may be enough.
• Moderate change rate (e.g., small business file server):
• Weekly full + daily incremental or differential.
• High change rate (e.g., database, transaction system):
• Daily full + frequent (e.g., hourly) incremental or log-based backups.
• Or continuous replication (beyond this chapter’s scope, but same planning ideas).

Consider:

• When is the system least busy?
  Run heavy backups off-peak.
• How large is the data?
  Large data may need smart scheduling, partial backups, or splitting across time.

---

Retention Policies

Retention is about how long you keep each backup.

Common patterns:

• Short-term retention
• Keep daily backups for 7–14 days.
• Medium-term
• Keep weekly backups for 4–8 weeks.
• Long-term / archival
• Keep monthly backups for 6–12+ months.
• Important for compliance, audits, or rare “we need last year’s version” cases.

You can express a policy like:

• Daily (incremental) for 14 days
• Weekly (full) for 8 weeks
• Monthly (full) for 12 months

You might implement this with naming conventions, directories, or your backup tool’s retention settings.

Trade-offs:

• More retention → more storage and management.
• Less retention → less safety if corruption is discovered late.

---

Storage Locations and Media Options

On-Site vs Off-Site

• On-site (local) backups
• Faster to create and restore.
• Vulnerable to local disasters and some security threats.
• Examples: external HDD, NAS, another server in the same rack.
• Off-site backups
• Slower but survive local disasters and many security issues.
• Examples: remote server (SSH, rsync), cloud storage, removable drives stored elsewhere.

Balanced strategy:

• Use local backups for fast restores.
• Use off-site backups for disaster recovery.

---

Common Backup Targets

• External hard drives
• Simple and affordable.
• Rotate at least two drives if possible (one off-site).
• Network shares (NFS/Samba) or backup servers
• Centralized storage for multiple machines.
• Requires network and server capacity.
• Cloud storage / object stores
• Off-site by design.
• Often pay-per-GB and per-transfer.
• Must handle encryption, API access, and cost control.

For each target, consider:

• Capacity and cost.
• Throughput (how long backups and restores will take).
• Security (encryption at rest and in flight).
• Reliability (RAID, snapshots, redundancy).

---

Designing a Practical Backup Plan

Example: Small Home Server

Requirements:

• Host runs a file server (photos, documents), a few services.
• Acceptable RPO: 1 day.
• Comfortable with some downtime during restore.

Possible strategy:

• What: /home, application data directories, important configs under /etc.
• Where:
• Local external HDD (connected to the server).
• Optional: encrypted remote snapshot to a cloud or rented VPS.
• How often:
• Daily incremental to external drive, weekly full.
• Weekly sync (or monthly, depending on bandwidth) to remote storage.
• Retention:
• Daily for 14 days, weekly for 8 weeks, monthly for 6–12 months.

---

Example: Small Business Web + Database Server

Requirements:

• Stores customer data; need strong protection.
• RPO: 1 hour for database, 4 hours for other data.
• RTO: few hours is acceptable.

Possible strategy:

• What:
• Databases (with proper, consistent dumps or snapshots).
• Web content, uploads, configs.
• Where:
• Local backup server in same office/rack.
• Off-site backup (cloud / second location).
• How often:
• Hourly database incremental/log backups.
• Daily file backups (incremental).
• Weekly full backups.
• Retention:
• Hourly DB logs for 48 hours.
• Daily backups for 30 days.
• Monthly archival for 1–3 years (depending on business requirements).

Here you might combine:

• Local snapshots or rsync to a backup server.
• Nightly or weekly tar archives pushed to encrypted remote storage.

---

Example: Developer Workstation

Requirements:

• Source code and configs crucial; many restore options via Git.
• RPO: a few hours for non-version-controlled data.
• User is technical and willing to manage tooling.

Possible strategy:

• What:
• Entire $HOME except bulky caches, plus selected config directories.
• Where:
• Local external drive (for quick full system restore).
• Encrypted remote backups (cloud or rented server).
• How often:
• Hourly snapshots of $HOME using filesystem snapshot tools (if available).
• Nightly sync to external drive.
• Daily or weekly encrypted backups to remote target.
• Retention:
• Short snapshot retention (1–3 days).
• Longer retention on external and remote backups (e.g., 30–90 days).

---

Automation and Scheduling

Even the best strategy fails if backups don’t actually run.

Key elements:

• Automation
• Use cron, systemd timers, or your backup tool’s scheduler.
• Arrange non-interactive commands (no passwords on the terminal).
• Monitoring
• Log backup results.
• Send notifications on failure (email, chat, monitoring system).
• Check backup size or runtime anomalies (sudden drop might mean data not included).

A basic approach:

1. Write a backup script that:
• Defines what to back up and where.
• Logs its actions and exit status.
2. Schedule it with cron or a systemd timer.
3. Periodically verify logs and monitor disk usage on backup targets.

---

Security Considerations

Backups often contain everything an attacker wants.

Points to plan:

• Encryption
• Encrypt backups, especially if stored off-site, on removable drives, or in the cloud.
• Consider key management: where keys are stored and who can access them.
• Access control
• Limit which accounts can read or write backup data.
• Limit login access to backup servers.
• Network security
• Use secure protocols (SSH, TLS).
• Avoid exposing backup services directly to the internet without protection.
• Data lifecycle
• When backups expire by retention policy, ensure they are properly deleted or media is securely wiped.

Remember: if backups are unencrypted and stolen, your security may be completely compromised.

Testing Restores

A backup that has never been tested is not reliable.

Elements of a restore test strategy:

• Regular test restores
• Pick some files/directories and restore them to a test location.
• Check that permissions, timestamps, and contents are correct.
• Periodic full-restore tests
• For critical systems, occasionally simulate a disaster:
• Restore to a test machine or VM.
• Confirm that services start and data is intact.
• Documented procedures
• Write down how to restore:
• Where backups are.
• Commands or tools needed.
• Order of steps (e.g., restore database before application).
• Keep this documentation accessible even if the primary system is down.

You can integrate test restores into your schedule (e.g., monthly or quarterly).

Versioning and Change Tracking

Backups naturally produce versions over time.

Plan for:

• How many past versions of a file you want to keep.
• How to find a specific version (by date, label, or snapshot ID).
• Handling large changes:
• Large binary files that change often can cause backups to grow quickly.
• You may need special handling (e.g., separate schedule, deduplication tools, or a different way to store them).

For developers or teams:

• Git and other VCS handle code history.
• Backups should complement this by protecting:
• Repositories themselves.
• Non-versioned project data (databases, assets, etc.).

---

Documenting Your Backup Strategy

Finally, an effective strategy is written down and kept up to date.

Your backup documentation should include:

• Systems and data covered.
• RPO and RTO assumptions.
• Types of backups (full, incremental, differential, snapshots).
• Schedules (when, how often).
• Storage locations and retention periods.
• Tools used and configuration locations.
• Restore procedures and who is responsible.

Even on a single personal machine, having a short text file with these points helps you stay consistent and remember how to restore when you’re stressed.

By defining clear RPO/RTO targets, choosing appropriate backup types and schedules, following guidelines like 3-2-1, and regularly testing restores, you create a backup strategy that is practical, robust, and tailored to your Linux environments.