KAHIBARO
28.3. Disaster Recovery
Table of Contents
Why Disaster Recovery Matters
Things will go wrong in production. Disks fail, bugs ship, regions go offline, someone runs DROP TABLE on the wrong database. Disaster recovery is about planning how your backend survives serious failures and how you bring it back online with minimal data loss and downtime.
If you do not plan this in advance, you will be forced to improvise under pressure, which usually leads to more data loss and longer outages.
Disaster recovery is mostly about strategy and process, not fancy tools. You will reuse concepts you already know, such as backups, replicas, and monitoring, and combine them into a coherent plan.
Key Concepts: RPO and RTO
Two metrics appear in every disaster recovery discussion:
- RPO (Recovery Point Objective)
How much data you can afford to lose, measured as time. - RTO (Recovery Time Objective)
How much downtime you can afford before the system is back.
Example:
- If your RPO is 5 minutes, you must be able to restore to a state no older than 5 minutes.
- If your RTO is 1 hour, users can tolerate up to 1 hour of downtime during an incident.
DR Core Metrics
- Recovery Point Objective (RPO): Maximum acceptable age of recovered data.
Example: RPO = 10 minutes. - Recovery Time Objective (RTO): Maximum acceptable time to restore service.
Example: RTO = 30 minutes.
Think in terms of trade-offs:
| Requirement | Rough RPO | Rough RTO | Typical cost / complexity |
|---|---|---|---|
| Personal blog | 24 hours | 24 hours | Low, daily backups |
| Small SaaS with paying customers | 5–15 minutes | 15–60 minutes | Moderate, frequent backups + replicas |
| Payment processing system | < 1 minute | A few minutes | High, multi-region, hot standby |
Your job is not to get RPO and RTO to zero. Your job is to match them to business needs at a reasonable cost and then design your system to meet them.
Types of Disasters
Disasters are not only region-wide outages. It is helpful to classify them, because mitigation strategies differ.
Logical vs Physical Disasters
Logical disasters are caused by data corruption or mistakes at the software or human level.
Examples:
- A buggy migration script truncates a table.
- An admin accidentally deletes an S3 bucket.
- A code bug overwrites all user emails with
null. - An attacker modifies or deletes data.
Physical disasters affect infrastructure.
Examples:
- Cloud region outage.
- Data center power loss or network failure.
- Disk or hardware failure on your database server.
- Loss of an entire Kubernetes cluster.
Logical disasters often require point-in-time recovery. Physical disasters often require redundant infrastructure.
Scope of Impact
You can also categorize by scope:
| Scope | Example | Typical mitigation |
|---|---|---|
| Single component | One DB instance fails | Replication, automatic failover |
| Single service | Payments service is broken | Graceful degradation, circuit breakers |
| Single region | Cloud region loses network | Multi-region deployment, DNS failover |
| Global | Cloud provider major outage, widespread DNS issue | Multi-cloud or accept downtime |
Do not over-engineer for global disasters if your system cannot yet tolerate a single database crash. Start with the most likely and most impactful failures.
Disaster Recovery Strategies
Disaster recovery strategies describe where your data and services exist and how you switch to a healthy copy.
Backup-Based Recovery
This is the simplest and most common strategy for many small and medium systems.
Process:
- Take regular backups of the primary database and critical storage.
- Store backups in a different location (for example, another region or bucket).
- When a disaster happens, restore from the latest good backup to a new instance.
- Point your application at the restored instance.
Pros:
- Simple to understand.
- Inexpensive compared to always-on replicas.
- Works for logical disasters if you keep enough history.
Cons:
- RPO is at least as large as the backup interval.
- RTO can be large, because restore may be slow.
- You must test restore procedures to avoid surprises.
Use this when:
- Data changes are not too frequent or critical.
- Some hours of potential data loss and downtime are acceptable.
Cold, Warm, and Hot Standby
For lower RPO and RTO, you use a standby system, which is a second environment that can take over.
Cold Standby
- A second environment is stopped or partially configured.
- You may have scripts or Terraform / Ansible to create servers, networks, etc.
- In a disaster, you deploy, restore backups, and switch traffic.
Characteristics:
- Low cost.
- RTO is high, because you must create and start resources.
- RPO depends on backup frequency.
Example: A secondary region where you only store periodic database backups and Terraform configs, but no running instances.
Warm Standby
- The secondary environment is running, but at reduced capacity.
- Database often runs as a replica of the primary (read-only).
- Application instances may be small or minimal.
In a disaster:
- Promote replica to primary.
- Scale application instances up.
- Switch traffic with DNS or a load balancer.
Characteristics:
- Higher cost than cold, lower than hot.
- RTO is moderate, typically minutes.
- RPO depends on replication lag, often seconds or minutes.
Hot Standby (Active-Passive or Active-Active)
Hot standby (active-passive):
- Secondary environment is fully running and replicated.
- It does not receive normal user traffic.
- Failover is quick, often automatic.
Active-active:
- Multiple environments serve production traffic at the same time (for example, multi-region).
- If one fails, the others continue to handle some or all traffic.
Characteristics:
- Highest cost and complexity.
- Lowest RTO.
- RPO close to zero in good setups.
Standby Environments
- Cold standby: Cheap, slow to recover.
- Warm standby: Medium cost, medium recovery time.
- Hot standby: Expensive, fast recovery, often near-zero RPO/RTO.
Multi-AZ and Multi-Region
Cloud providers often give you availability zones (AZs) inside a region and multiple regions across geographic areas.
- Multi-AZ: Deploying your database and application across multiple AZs in the same region.
- Multi-region: Deploying in multiple geographic regions.
Typical pattern:
- Use multi-AZ for high availability inside a region. This protects from many physical failures at low complexity.
- Consider multi-region when:
- RTO and RPO requirements are very strict.
- Compliance requires regional diversity.
- You serve global users and need low latency.
Multi-region often implies:
- Replicated databases across regions.
- DNS-based routing (like latency-based or failover routing).
- Careful handling of global state and data conflicts.
Designing a Disaster Recovery Plan
A disaster recovery plan is a documented, testable set of procedures that explain exactly what to do when a disaster happens.
Identify Critical Components and Data
List your system components and decide which are critical to minimal service.
Examples:
| Component | Is critical? | Reason |
|---|---|---|
| Primary PostgreSQL DB | Yes | Contains all user and business data |
| Redis cache | No | Can be rebuilt from database |
| Authentication service | Yes | Without it, no one can log in |
| Email sending service | Maybe | Might be delay-tolerant in some systems |
| Metrics dashboard | No | Helpful but not required to serve traffic |
Also identify critical data stores:
- Relational databases (PostgreSQL, MySQL).
- NoSQL databases (Redis when used as primary storage, MongoDB).
- Object storage (S3 buckets with user uploads).
- Configuration and secrets stores (Vault, SSM, environment variables).
- Queues that store important messages (RabbitMQ, Kafka).
You only need full DR for critical components. Others can use simpler mitigation, such as recreating from code or resynchronizing later.
Define RPO and RTO per Component
For each critical component, define:
- Target RPO (how much data loss is ok).
- Target RTO (how long outage is ok).
Example table:
| Component | RPO target | RTO target | Notes |
|---|---|---|---|
| PostgreSQL DB | 5 minutes | 30 minutes | Frequent backups, point-in-time recovery |
| Main API service | 0 minutes | 15 minutes | Redeploy from CI, multi-AZ load balancing |
| S3 uploads | 24 hours | 24 hours | Versioned bucket, cross-region replica |
| Auth tokens store | 10 minutes | 30 minutes | Re-issue tokens if needed |
Once you have numbers, you can design backups and failover mechanisms to meet them.
Document Recovery Procedures
Your plan must describe step-by-step procedures for common disaster scenarios. Anyone on the team should be able to follow them.
Example for "Primary DB instance destroyed in region A":
- Confirm that the DB is unavailable and not simply slow.
- Decide whether to fail over to a replica or restore from backup.
- If using a replica:
- Promote replica in region B to primary.
- Update DB connection string in configuration / secret store.
- Restart application instances with new configuration.
- If restoring from backup:
- Provision a new DB instance in region B.
- Restore from the latest full backup and replay WAL logs up to target time.
- Run schema migrations if needed.
- Update configuration and restart application.
- Verify application can read and write.
- Announce partial or full service restoration.
Write these steps in a shared document or runbook and keep them updated as the system evolves.
Data Backup Strategies for DR
Disaster recovery is impossible without reliable backups. Backups are your last line of defense against many disasters, especially logical ones.
Types of Backups
The three common backup types:
| Type | Description | Pros | Cons |
|---|---|---|---|
| Full | Complete copy of data at a point in time | Simple, easiest to restore | Large, slow to create |
| Incremental | Changes since the last backup (any type) | Smaller, fast to create | Restore may require many backups |
| Differential | Changes since the last full backup | Balance between full and incremental | Grows until next full backup |
For databases like PostgreSQL, you often combine:
- Base backup (similar to a full backup).
- WAL (write-ahead log) archiving for continuous changes.
This allows point-in-time recovery: you restore base backup and replay logs up to a specific timestamp.
Backup Frequency and Retention
To meet your RPO, choose a suitable backup frequency.
Example:
- RPO = 4 hours: Full backup every night, incremental every 4 hours.
- RPO = 15 minutes: Base backup daily, WAL archived continuously.
You also need a retention policy:
- How long to keep daily backups? Weekly? Monthly?
- When to delete old backups?
Typical pattern:
- Keep daily backups for 7–14 days.
- Keep weekly backups for a few months.
- Keep monthly backups for 6–12 months, depending on compliance.
Backup Rules
- Always store backups outside the primary failure domain (for example, different region or account).
- Treat backups as sensitive data, protect them with access control and encryption.
- Test restoring from backups regularly. A backup that cannot be restored is useless.
Where to Store Backups
Options include:
- Object storage such as S3, GCS, Azure Blob.
- Different region or availability zone.
- In another cloud account or project for extra isolation.
- For critical systems, another provider entirely.
At minimum:
- Do not keep your only backups on the same disk or same database server.
- Use object storage in a different AZ or region, and ideally a different account.
Failover and Failback
Backups let you rebuild, but failover handles switching production traffic to a healthy environment. Failback is returning to normal once the disaster is over.
Failover Mechanisms
Common techniques:
- Database failover
- Promote a read replica to primary.
- Use DNS names or configuration variables so the app does not reference IPs directly.
- Some managed databases support automatic failover, but you still need to plan for it.
- Application failover
- Run application instances in multiple AZs.
- Use a load balancer that can stop sending traffic to unhealthy instances.
- For multi-region, use DNS-level failover (for example, health-check based).
- Storage failover
- Use cross-region replicated buckets.
- If the primary region is down, switch configuration to use the secondary bucket.
Failover must be:
- Predictable: clearly defined steps or automation.
- Safe: avoid split-brain situations, where two primaries write to the same data independently.
- Observable: you must know when failover succeeded.
DNS and Load Balancing in DR
DNS often plays a central role in DR.
Pattern:
- You have a domain like
api.example.com. - In normal operation, it points to a load balancer in Region A.
- In a disaster, you change DNS to point to Region B.
Things to consider:
- DNS TTL (time to live) affects how quickly clients see the new target.
- Short TTL (for example, 60 seconds) gives faster switch, but more frequent DNS lookups.
- Some managed DNS services support health checks and automatic failover.
- Internally, your app might use service discovery or internal DNS names that you can update.
Test at least:
- Manual DNS cutover.
- How long it takes for most traffic to shift.
Failback Considerations
Once the original region or system is healthy, you may want to move back.
Challenges:
- During failover, data is written to the new primary. You must ensure those writes are safely replicated back to the original location.
- You must avoid diverging histories. Plan carefully for how replication flows during and after failover.
Typical steps:
- After crisis, treat the new primary as the source of truth.
- Rebuild the old primary from the new primary, either:
- Through replication, or
- By taking a fresh backup and restoring it.
- Once old primary is synchronized and acts as a replica, you can optionally promote it and redirect traffic again.
Sometimes, you decide not to fail back at all. You keep the new environment as the permanent primary.
Testing and Practicing Disaster Recovery
A disaster recovery plan that nobody has practiced is only a guess. Regular testing is crucial.
DR Drills and Game Days
You can schedule DR drills, sometimes called game days, to simulate disasters.
Examples:
- Simulate primary database loss:
- Turn off or isolate the main DB (in a test or staging environment).
- Follow your DR runbook to fail over to the standby.
- Simulate accidental data deletion:
- Delete or corrupt a table in a test database.
- Restore from backup and measure how long it takes.
Practical tips:
- Start with drills on staging, then later use limited, controlled experiments in production.
- Involve multiple team members, not just a single expert.
- Time each step and note where confusion or delays occur.
Measuring and Improving RPO/RTO
During drills, measure:
- Time between last valid data point and restored data point (actual RPO).
- Time from incident start to full service restoration (actual RTO).
If actual values are worse than targets, you have two options:
- Improve the system:
- More frequent backups.
- Faster restores or automated failover.
- Better tooling and scripts.
- Or adjust the targets:
- Talk with stakeholders and adapt expectations to what is realistic and cost-effective.
Also track:
- How many manual steps are required.
- How often human error occurs during DR drills.
Your goal is to reduce manual, error-prone steps over time.
DR for Databases, Files, and Services
Different components need different DR techniques.
Databases
For relational databases such as PostgreSQL:
Common patterns:
- Automated backups (full + WAL) with retention and cross-region storage.
- Read replicas in another AZ or region.
- Point-in-time recovery to handle logical corruption.
In DR:
- For logical corruption:
- Determine when corruption began.
- Restore from backup before that time.
- Replay WAL logs only up to right before corruption.
- For physical loss of primary:
- Promote an existing replica.
- Or restore latest backup to a new instance.
Make sure schema migrations are:
- Stored in code.
- Idempotent or well-managed.
- Part of your restore process.
File Storage and Object Storage
User uploads such as images, documents, and logs often live in object storage.
Recommended practices:
- Use versioned buckets where possible. This helps undo accidental deletions.
- Use cross-region replication for critical data.
- Maintain a mapping of files in your database, so you can check consistency.
In DR:
- If a bucket is lost or corrupted, restore from:
- Snapshot or cross-region replica.
- Separate backup bucket.
Note that large volumes of files can make restore slow. In many systems, user experience tolerates missing very old files better than missing recent database records.
State in Caches, Queues, and Other Services
Caches:
- Typically do not need strong DR. You can rebuild from the database.
- Focus on having a plan to restart or replace them quickly.
Message queues:
- If queues hold critical messages that cannot be lost, use:
- Durable queues.
- Replication and backups.
- DR must ensure:
- Messages are not lost.
- You do not process messages twice without handling idempotency.
Other services such as search indexes:
- You can often rebuild from the primary database.
- DR plan usually describes:
- How to recreate the index.
- How long it is acceptable for search to be degraded.
Human and Process Aspects
Disaster recovery is also about people, communication, and decision making.
Roles and Responsibilities
Define who does what during an incident:
Possible roles:
- Incident commander: Person who leads the response, makes final decisions, and coordinates others.
- Technical lead: Person who decides which technical steps to take.
- Communicator: Person who updates stakeholders and possibly customers.
- Scribe: Person who records timeline and actions for later review.
You do not need a large team, but you do need clarity. Even in a small startup, you can decide that:
- The on-call engineer is incident commander.
- The most experienced backend engineer is technical lead.
- A product manager handles communication.
Communication During a Disaster
While you are recovering, users and internal stakeholders must know what is happening.
Typical communication items:
- A short description of the issue.
- Which services are impacted.
- Whether data might be lost or inconsistent.
- Estimated time to recovery, even if very rough.
- Progress updates at regular intervals.
Avoid being overly optimistic. State clearly when you do not know yet and will update later.
After the incident, write a post-incident report. It should describe:
- What happened.
- Impact on users and data.
- Root cause or contributing factors.
- What went well.
- What will be improved, including DR plan changes.
Putting It All Together
For a typical backend in this course context, a practical, minimal disaster recovery setup might look like this:
- PostgreSQL:
- Automated daily full backups.
- WAL archiving or frequent incremental backups.
- At least one read replica in another AZ or region.
- Object storage:
- Versioned bucket for user files.
- Cross-region replication for critical uploads.
- Application:
- Deployed in multiple AZs behind a load balancer.
- Configuration and secrets stored in a central, backed-up location.
- DNS:
- Short TTL.
- Documented procedure to reroute traffic to a backup region.
- DR Plan:
- Documented steps for database failure, region outage, and major data corruption.
- RPO/RTO targets defined and realistic.
- DR drill at least a few times per year.
By combining solid backups, clear failover paths, and regular practice, you turn disasters from existential threats into difficult but manageable events.
Views: 8