KAHIBARO
Discord Login Register ☰
☰

24.13. Rollback Strategies

Table of Contents

Why Rollback Strategies Matter

In CI/CD, deployments are frequent and automated. Eventually, a deployment will break something. A rollback strategy is your safety net, a clear and tested way to get your system back to a healthy state fast.

Without a rollback strategy you might:

With a rollback strategy you:

A deployment without a rollback strategy is not a safe deployment.

This chapter explains the main rollback strategies, when to use them, and how they relate to CI/CD pipelines.

Concepts: Rollback vs Roll-Forward

Before looking at specific strategies you need two core ideas.

Rollback

A rollback means returning your system to a previous known good version.

Examples:

Characteristics:

Roll-Forward

A roll-forward means deploying a new fix that corrects the problem in the latest version.

Examples:

Characteristics:

In CI/CD you usually:

  1. Rollback quickly to restore service.
  2. Then roll-forward with a fix after proper testing.

Simple Rollback: Redeploy Previous Version

The simplest rollback is to redeploy the last working build.

Requirements

To do this safely your pipeline must:

Example of versioned Docker images:

VersionDocker TagStatus
1.4.0app:1.4.0Stable
1.5.0app:1.5.0Broken
latestapp:latestPoints to 1.5.0

If 1.5.0 is broken, you run:

bash 
docker service update \
  --image my-registry.com/app:1.4.0 \
  my-app-service

In Kubernetes, you might change the image tag in your deployment manifest and apply it again.

Integrating with CI/CD

In a simple CI/CD pipeline:

Pipeline example (pseudo YAML):

yaml 
jobs:
  deploy_prod:
    steps:
      - name: Deploy current version
        run: ./deploy.sh $CI_COMMIT_SHA
  rollback_prod:
    when: manual
    steps:
      - name: Get last stable version
        run: ./get_last_stable.sh > version.txt
      - name: Deploy last stable version
        run: ./deploy.sh $(cat version.txt)

Pros and Cons

ProsCons
Simple to understand and implementMay not handle DB schema changes well
Works with many hosting environmentsUsers might lose access to new features
Easy to automate in CI/CDData shape must remain backward compatible

For simple redeploy rollbacks, always keep at least one previous artifact and track which version is stable.

Database-Aware Rollbacks

Code rollbacks are usually quick. Database rollbacks are trickier.

When you deploy a new version, you often apply migrations that change the database schema. For example:

sql 
ALTER TABLE users ADD COLUMN phone_number TEXT;

If you later roll back the code, you must ensure the database schema and data are still compatible.

Forward-Only Migrations vs Down Migrations

With migration tools like Alembic, Liquibase, or Flyway you can:

But not all changes are easy to roll back. For example:

sql 
ALTER TABLE orders DROP COLUMN notes;

If you drop a column, you lose the data, and a down migration cannot magically restore it.

Because of this, many teams practice forward-only migrations:

Designing Backward-Compatible Changes

To make rollbacks safe, design database changes that are backward compatible with the previous code version.

Common patterns:

  1. Add, then use, then remove:
    • Deployment 1: Add new column new_price, keep old price.
    • Deployment 2: Start writing to both price and new_price.
    • Deployment 3: Read from new_price only.
    • Deployment 4: Drop old price.
  2. Soft changes before hard changes:
    • Add constraints, but do not enforce strictly at first.
    • Clean data.
    • Enforce constraints later.
  3. Avoid destructive changes during risky features:
    • Do not drop tables or columns in the same release that introduces complex feature changes.

Database Backups as a Rollback Option

In serious failures, you might need to restore from backup.

Typical sequence:

  1. Deployment runs a wrong migration that deletes important data.
  2. Application breaks, or data is corrupted.
  3. Incident response:
    • Stop write traffic (maybe switch to maintenance mode).
    • Restore database from a backup taken just before the migration.
    • Redeploy the last stable application version.

Restoring from backup can be slow and might lose recent data, so it is a last resort rollback.

Never rely only on backups for routine rollbacks. Use backups for disaster recovery, not for normal feature rollbacks.

Canary Releases and Progressive Rollback

A canary release means you deploy the new version to a small part of traffic first.

If it behaves well, you gradually increase the percentage of traffic. If it fails, you roll back to the previous version before everyone is affected.

Example Traffic Progression

StepNew Version TrafficOld Version Traffic
15%95%
225%75%
350%50%
4100%0%

If error rates spike at step 2, you can immediately send traffic back to the old version.

CI/CD Integration

A typical canary pipeline:

  1. Deploy new version as a separate instance (or set of instances).
  2. Update routing (load balancer or service mesh) to send 5 percent traffic to new version.
  3. Monitor metrics and logs for a defined time, for example 15 minutes.
  4. If OK, increase to 25 percent, then 50 percent, then 100 percent.
  5. If not OK at any step:
    • Send traffic back to 0 percent new version.
    • Mark deployment as failed.

You can automate this with:

Rollback in Canary

Rollback here is mainly about traffic routing:

This is usually faster and safer than redeploying code since both versions are already running.

Blue-Green Deployments and Instant Rollback

A blue-green deployment keeps two full environments:

At any time, traffic flows to exactly one environment.

How It Works

  1. Blue environment runs version 1.4.0.
  2. CI/CD pipeline deploys version 1.5.0 to Green environment.
  3. You run tests and health checks against Green.
  4. When ready, switch traffic from Blue to Green.
  5. If there is a problem:
    • Switch traffic back from Green to Blue.

Traffic switching happens at route level:

Rollback with Blue-Green

Rollback is simply:

The old version is still running and ready. This makes rollback instant.

Example using DNS or load balancer:

Stateapi.example.com Points To
Before deploymentBlue environment
After successful testGreen environment
After failureBack to Blue environment

Considerations

Blue-green is very powerful, but:

In blue-green deployment, never perform destructive DB changes that only work with the new version. Design migrations to keep both blue and green versions working during the switch.

Feature Flags as a Logical Rollback

Feature flags (also called feature toggles) allow you to enable or disable specific code paths without redeploying.

Instead of rolling back the whole application, you can turn off just the failing feature.

Basic Idea

In code:

python 
if is_feature_enabled("new_checkout"):
    run_new_checkout()
else:
    run_old_checkout()

Configuration system:

Rollback with Feature Flags

Timeline:

  1. You deploy version 1.5.0, which contains new checkout code guarded by flag new_checkout.
  2. Initially, new_checkout = false, so only old checkout runs.
  3. You slowly turn new_checkout to true for, say, 10 percent of users.
  4. If you detect problems with new checkout:
    • Set new_checkout = false.
    • All users return to old checkout behavior without changing code.

In many systems, feature flags are updated instantly via:

This is a logical rollback:

Combining Feature Flags with CI/CD

In CI/CD pipelines:

This reduces the need to roll back entire releases, because you can limit problematic features quickly.

Automatic vs Manual Rollbacks

Not every rollback should be fully automatic, but some can be.

Automatic Rollbacks

An automatic rollback triggers based on objective signals:

Example rule:

In CI/CD this can be integrated with:

Manual Rollbacks

Manual rollbacks are triggered by a human:

Best practice is to:

Example:

Humans should decide when to roll back, but scripts and tools should execute the rollback. Avoid manual server changes during incidents.

Versioning and Traceability

Rollback strategies only work if you know exactly what you are rolling back to.

You need:

Example deployment record table:

EnvironmentCurrent VersionPrevious VersionDeployed At
staginga1b2c3d789abcd2026-08-28 10:15 UTC
prod789abcd12345672026-08-28 09:00 UTC

If production breaks, you immediately see that rolling back means deploying 1234567.

Choosing a Rollback Strategy

You rarely use only one strategy. Typically you combine several:

ContextGood Strategy Combination
Small app, simple DBRedeploy previous version, basic DB backups
Growing app, relational DBBackward-compatible DB migrations, redeploy, manual rollback job
Larger system, high uptime requirementBlue-green or canary, feature flags, automatic rollback
Many teams, many featuresFeature flags for most new features, canary release for risky ones

Guidelines:

  1. Always keep the ability to redeploy an older version.
  2. Make database migrations safe to roll back or forward.
  3. For user-facing, critical systems, use canary or blue-green.
  4. Use feature flags for big or risky changes.
  5. Automate the rollback action inside your CI/CD platform.

Practicing Rollbacks

Rollback strategies are only useful if they actually work in real life.

Practice them:

You can even write automated tests for your deployment and rollback scripts, for example:

A rollback strategy that has never been tested is not a real strategy. Treat rollbacks as a core part of your CI/CD process and test them regularly.

By building and practicing clear rollback strategies, you turn deployments from risky events into routine, reversible operations, which is one of the main goals of CI/CD.

Views: 6

Comments

Please login to add a comment.

Don't have an account? Register now!

© 2025 Kahibaro
Visitors: 2998 / 24h
Impressum / Imprint
Datenschutz / Data Policy