KAHIBARO
Discord Login Register ☰
☰

23.11. Rollbacks

Table of Contents

Why Rollbacks Matter

Even with careful testing, some deployments will break things. A rollback is how you safely go back to a previous, known good state.

Typical reasons you need a rollback:

A good deployment strategy is incomplete without a clear, practiced rollback plan. If you cannot revert quickly, you do not really control your production system.

In this chapter you will see practical rollback patterns that fit modern Docker and CI/CD based backends.

Rollback Principles

Separate “deploying” from “releasing”

A useful mental model is:

If you separate these two, rollbacks are easier:

This is why strategies like blue green and canary are so powerful.

Make rollbacks fast and predictable

A rollback should be:

Examples of “bad” rollback situations:

Treat rollback as a first-class use case

When you design deployment scripts, Docker images, and CI/CD pipelines, always ask:

For every new deployment feature, add a rollback step to your mental checklist.

Versioning and Immutable Artifacts

Why versions matter for rollback

You can only rollback to what you can uniquely identify and reproduce. That means you must have clear versioning of:

Good versioning allows you to say: “Production is currently running app:1.4.2. We want to rollback to app:1.4.1.”

Common versioning strategies

ItemExample versionNotes
Docker image tagmyapp:1.4.2, myapp:2024-08-18Avoid using only latest in production.
Git commita1b2c3dCI can attach commit SHA to image tag.
Release namev1.4.2Semantic versioning is popular.
Migration batch202408181230_add_orders_tableTimestamp-based migration ids.

Rule: Never deploy something you cannot identify and reproduce.
Each production deployment must be tied to a specific, immutable artifact (image + commit + migrations).

Immutable artifacts

An immutable artifact is built once and never changed. For Docker-based apps:

This gives you:

Example Docker tags in a registry:

TagUse case
myapp:1.4.2Immutable release version
myapp:1.4.2-a1b2c3dVersion + commit
myapp:stagingMoving tag pointing at current staging
myapp:prodMoving tag pointing at current prod

Rollback then becomes:

Deployment Strategies and Rollbacks

Your deployment strategy strongly affects how easy a rollback is.

In-place deployments

In-place means you update the same servers or containers in place:

  1. Stop service
  2. Replace code or container image
  3. Start service again

This is simple but has weak rollback support:

Use in-place with great care for:

If you have to use in-place, keep a “previous version” deployment script ready:

bash 
# Bad: only deploys "latest"
docker pull myapp:latest
docker stop myapp || true
docker rm myapp || true
docker run -d --name myapp myapp:latest

Better:

bash 
# Good: explicit versions
# rollout
docker pull myapp:1.4.2
docker stop myapp || true
docker rm myapp || true
docker run -d --name myapp myapp:1.4.2
# rollback script (prepared ahead of time)
docker pull myapp:1.4.1
docker stop myapp || true
docker rm myapp || true
docker run -d --name myapp myapp:1.4.1

Blue green deployment

With blue green, you have two identical environments:

Process:

  1. Deploy new version to green.
  2. Run tests and checks on green.
  3. Switch traffic from blue to green, usually by:
    • Changing load balancer config
    • Updating a reverse proxy (Nginx, Traefik)
  4. Keep blue running as backup for some time.

Rollback is easy:

Example concept with Nginx upstreams:

nginx 
upstream app_backend {
    server blue.example.internal;   # old version
    # server green.example.internal; # new version
}

To switch:

nginx 
upstream app_backend {
    # server blue.example.internal;   # old version
    server green.example.internal;   # new version
}

Rolling back is just switching back the commented line and reloading Nginx.

Advantages:

Trade-offs:

Canary releases

Canary deployment sends a small percentage of traffic to the new version first.

Example steps:

  1. Deploy new version to a subset of servers or a separate service.
  2. Send 1% of traffic to new version, 99% to old.
  3. Monitor errors and performance.
  4. If things look good, increase to 10%, then 50%, then 100%.

Rollback strategy:

This is very useful when:

Implementation often uses:

You do not need all that complexity on day one, but remember the concept.

Rolling updates

Rolling update replaces instances one by one:

  1. You have N instances of your app.
  2. Deployment system stops instance 1, updates it, brings it back.
  3. Repeats for instances 2, 3, ..., N.

This is common in systems like Kubernetes.

Rolling rollback typically:

If you use a rolling strategy, make sure your platform supports:

Database Changes and Rollbacks

Application rollbacks are usually easy compared to database rollbacks.

The big risk: incompatible schemas

Example scenario:

  1. Old version expects column users.username.
  2. New version:
    • Renames username to user_name.
    • Deploys new code that uses user_name.
  3. Migration drops username column.

If you deploy new code and migration together, then realize there is a bug:

You now have an application and database mismatch.

Rule: Design migrations to be backward compatible when possible.
The old application version must still work with the new schema during and after deployment.

Safe migration patterns

General safe strategy:

  1. Add, do not remove:
    • Add new columns or tables.
    • Keep old columns for now.
  2. Deploy code that uses the new structure:
    • Or code that supports both old and new.
  3. Migrate data:
    • Backfill new columns from old ones.
  4. Wait:
    • Make sure the new version is stable.
  5. Remove old structures in a later deployment:
    • Only after you are sure you will not roll back to old code.

Example for renaming a column:

  1. Migration 1:
    • Add user_name column.
  2. Deploy version 1.4.0:
    • Write both username and user_name.
    • Read user_name if present, otherwise username.
  3. Run a background job to copy username to user_name for all users.
  4. After some time and a few releases:
    • Deploy version 1.5.0 that only reads user_name.
  5. Migration 2:
    • Drop username column.

Now if you rollback from 1.5.0 to 1.4.0:

Migration tools and rollback

Most backends use a migration tool, for example:

These tools track:

You can usually:

But downgrades are not always safe, especially if:

Practical advice:

Forward-only fix example:

Rollback Procedures

You need explicit procedures that your team can follow, step by step.

Simple Docker-based rollback

Imagine a single-node Docker deployment with Nginx as reverse proxy. Production currently runs myapp:1.4.1.

Deployment to 1.4.2:

bash 
# Pull new image
docker pull myapp:1.4.2
# Stop old container
docker stop myapp || true
docker rm myapp || true
# Run new version
docker run -d \
  --name myapp \
  --network my_network \
  -e APP_ENV=production \
  myapp:1.4.2

Rollback to 1.4.1:

bash 
docker pull myapp:1.4.1
docker stop myapp || true
docker rm myapp || true
docker run -d \
  --name myapp \
  --network my_network \
  -e APP_ENV=production \
  myapp:1.4.1

To make this safer:

Docker Compose rollback

With docker-compose.yml, you might manage versions via image tags.

Example docker-compose.yml snippet:

yaml 
services:
  app:
    image: myapp:1.4.2   # current
    environment:
      - APP_ENV=production
    depends_on:
      - db

To rollback to 1.4.1:

  1. Edit the file:
yaml 
services:
  app:
    image: myapp:1.4.1   # rollback target
  1. Run:
bash 
docker compose pull app
docker compose up -d app

Some teams keep:

The rollback procedure is then:

  1. git revert to the commit with previous working docker-compose.prod.yml
  2. docker compose up -d

Kubernetes style rollback (conceptual)

If you deploy to Kubernetes, you typically use a Deployment resource. Kubernetes automatically tracks rollout history.

Deployment excerpt:

yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 3
  template:
    spec:
      containers:
        - name: app
          image: myapp:1.4.2

Upgrade and rollback:

bash 
# Apply new deployment (1.4.2)
kubectl apply -f deployment.yaml
# If something goes wrong:
kubectl rollout undo deployment/myapp

Kubernetes tracks the last few ReplicaSet versions. rollout undo switches back to the previous one.

Even if you do not use Kubernetes yet, it is useful to understand this pattern, because many modern systems work similarly.

CI/CD pipeline rollbacks

Your CI/CD system can provide “promote” and “rollback” jobs.

Example conceptual pipeline steps:

StepDescription
buildBuild Docker image, run tests.
deploy_stagingDeploy to staging.
deploy_productionDeploy image myapp:1.4.2 to prod.
rollback_productionDeploy previous tag myapp:1.4.1 to prod.

Rollback job might:

Key ideas:

Testing and Practicing Rollbacks

A rollback strategy only works if it is:

Test rollbacks in non-production environments

You can regularly practice:

  1. Deploy version A to staging.
  2. Deploy version B to staging.
  3. Intentionally rollback to version A in staging.
  4. Verify:
    • App runs correctly.
    • Database is consistent.
    • Logs and monitoring look normal.

This reveals:

Run through failure scenarios

Design some simple “what if” scenarios and verify that your rollback procedure works:

ScenarioCheck
New version fails to startCan you quickly restart old version?
Increased error rate after 5 minutesCan you switch traffic back fast?
Background worker starts corrupting dataCan you stop workers and rollback safely?
Bad migration drops an important columnDo you have backups and restore procedures?

Document clearly:

Post-rollback actions

After you rollback a broken release:

  1. Freeze new deployments temporarily.
  2. Collect:
    • Logs
    • Metrics
    • Error reports
  3. Create an incident report:
    • What went wrong?
    • How did we detect it?
    • What made rollback hard or easy?
  4. Improve:
    • Tests
    • Monitoring
    • Deployment / rollback scripts
    • Migration patterns

The goal is that each rollback makes your system safer for the next deployment.

Summary

Key ideas from this chapter:

With a solid rollback strategy, you can deploy more often and with more confidence, because if something goes wrong you know exactly how to go back.

Views: 7

Comments

Please login to add a comment.

Don't have an account? Register now!

© 2025 Kahibaro
Visitors: 3096 / 24h
Impressum / Imprint
Datenschutz / Data Policy