KAHIBARO
Discord Login Register ☰
☰

23.10. Zero-Downtime Deployment

Table of Contents

Why Zero-Downtime Deployment Matters

When you deploy a new version of your backend, you usually restart processes, migrate databases, and update configuration. If you do this in a naive way, the application becomes unavailable for a short period. Users see errors, timeouts, or failed requests.

Zero-downtime deployment is about deploying new versions without interrupting existing traffic. Requests continue to succeed while the new version is rolled out, and users should not notice any disruption.

You care about this when:

Typical problems you want to avoid:

In this chapter, you will see common patterns and practical techniques to keep your backend online during deployments.

Core Strategies for Zero-Downtime

There are many tools and platforms, but most zero-downtime strategies are built from a small set of ideas.

Blue-Green Deployments

In a blue-green deployment, you maintain two production environments:

When the green version is ready:

  1. Route traffic from blue to green, usually by updating:
    • A load balancer configuration.
    • A reverse proxy configuration.
    • A DNS record, in some setups.
  2. Keep the blue environment running for a while:
    • If something goes wrong, you switch traffic back to blue quickly.

A simplified flow:

  1. Current production: v1 on environment blue.
  2. Deploy v2 to environment green, but do not send user traffic yet.
  3. Run smoke tests against green (health checks, basic API tests).
  4. Switch traffic to green.
  5. Monitor errors and metrics.
  6. If stable, decommission blue or prepare it to host the next deployment.

Advantages:

Disadvantages:

Rolling Deployments

With a rolling deployment, you update your application instances gradually. At any moment, some instances run the old version and some run the new version.

For example, imagine you have 4 application containers behind a load balancer:

  1. Mark container 1 as unavailable in the load balancer.
  2. Stop container 1, start new version, wait for it to be healthy.
  3. Add container 1 back to the load balancer.
  4. Repeat for containers 2, 3, and 4.

During the rollout:

Advantages:

Disadvantages:

Canary Releases

A canary release sends only a small amount of traffic to the new version at first.

Example:

  1. 95% of traffic goes to version v1.
  2. 5% goes to new version v2 (the canary).
  3. You compare metrics:
    • Error rate.
    • Latency.
    • Business metrics like signup success.
  4. If metrics look good, gradually increase v2 traffic:
    • 10%, 25%, 50%, 100%.
  5. If something is wrong, send all traffic back to v1.

This is similar to blue-green, but the switch is gradual, not all at once.

Canary releases are often implemented with:

In-Place Restart with Graceful Shutdown

Sometimes you do not have advanced infrastructure, but you can still reduce downtime by:

You can combine:

This approach is often the simplest path to "almost zero" downtime for small setups.

Load Balancers and Traffic Switching

Load balancers are central to zero-downtime deployments. They control where traffic goes, which lets you add or remove backend instances without breaking clients.

Using a Reverse Proxy as a Load Balancer

For many small to medium systems, you can use Nginx as:

A basic Nginx upstream with two FastAPI/Uvicorn instances:

nginx 
upstream myapp {
    server 127.0.0.1:8001;
    server 127.0.0.1:8002;
}
server {
    listen 80;
    location / {
        proxy_pass http://myapp;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

To update with near zero downtime:

  1. Start new instances on different ports.
  2. Add them to the upstream block.
  3. Reload Nginx with nginx -s reload (reload is graceful).
  4. Remove old instances when they are no longer used.

You can implement a simple rolling update by:

Health Checks

Health checks tell the load balancer which instances are healthy.

Typical patterns:

Example of a simple FastAPI health check:

python 
from fastapi import FastAPI
app = FastAPI()
@app.get("/health")
def health():
    return {"status": "ok"}

Health checks are critical during deployments:

Traffic Weighting and Routing

More advanced systems let you control traffic by weights. For example:

This means that ~90% of requests go to v1 and ~10% to v2.

You can also route by:

This routing flexibility is used in canary releases and A/B testing.

Graceful Shutdown and Draining Connections

Your backend must support graceful shutdown, or you risk killing in-flight requests and causing errors during deployment.

What Is Graceful Shutdown?

During a graceful shutdown:

  1. Your process stops accepting new connections.
  2. It waits for active requests to complete.
  3. It releases resources, such as:
    • Database connections.
    • Message queue connections.
    • File handles.
  4. It then exits cleanly.

Without graceful shutdown, deployment tools or system managers may send a kill signal that stops the process immediately and drops active requests.

Signals and Shutdown in Python Servers

Many servers respond to POSIX signals:

Uvicorn, which you commonly use with FastAPI, already supports graceful shutdown by default. When it receives SIGTERM or SIGINT, it:

You can customize startup and shutdown hooks in FastAPI:

python 
from fastapi import FastAPI
app = FastAPI()
@app.on_event("startup")
async def on_startup():
    # Initialize database connections, caches, etc.
    ...
@app.on_event("shutdown")
async def on_shutdown():
    # Close database connections, flush logs, etc.
    ...

These hooks are run for each process when it starts and stops, which is very helpful for graceful deployments.

Connection Draining

Connection draining is the process of gradually removing an instance from service:

  1. Mark instance as "draining" in the load balancer:
    • No new connections are sent to it.
    • Existing connections are allowed to continue.
  2. Wait for active requests to complete, up to some timeout.
  3. Stop the application instance.

You can implement this with:

If you forget draining, you may kill in-flight requests and cause user-visible errors.

Always combine deployments with graceful shutdown and connection draining.
Killing processes abruptly during deployment can lead to failed API calls, partial writes, and user-facing errors.

Database Migrations Without Downtime

Even if your application processes are deployed perfectly, database changes can break zero-downtime if not planned.

Problem cases:

To achieve zero-downtime, your schema migrations must be backward compatible with both old and new versions of the code, at least during the transition.

Backward Compatible Changes

A change is backward compatible if the old version of your app still works correctly after the change. Common safe changes:

Unsafe changes that require extra care:

Expand and Contract Pattern

A very common approach is often called "expand and contract".

  1. Expand phase: make schema changes that support both old and new code.
  2. Deploy new code that uses the new schema.
  3. Contract phase: remove old parts of the schema that are no longer used.

Example: renaming a column full_name to name.

Naive approach (bad for zero-downtime):

  1. Run migration: rename full_name to name.
  2. Deploy code that expects name.

If an instance still runs old code, it will break because full_name no longer exists.

Expand and contract approach:

  1. Expand: add a new column name and keep full_name:
sql 
   ALTER TABLE users
       ADD COLUMN name TEXT;
  1. Deploy code version v2 that:
    • Writes to both full_name and name.
    • Reads from name if available, otherwise falls back to full_name.
  2. Run a background job or one-time script that copies:
sql 
   UPDATE users
   SET name = full_name
   WHERE name IS NULL AND full_name IS NOT NULL;
  1. Once you are sure all running code handles name, deploy v3 that:
    • Reads only name.
    • Writes only name.
  2. Contract: drop the old column:
sql 
   ALTER TABLE users
       DROP COLUMN full_name;

At each step, both old and new versions of the code, and the database, can coexist without breaking.

Rule for zero-downtime database changes:
Each migration step must be safe to run while both the old and new versions of the application are live at the same time.

Long-Running Migrations

Some migrations can take a long time, for example:

Long migrations can:

To reduce risk:

For PostgreSQL, prefer operations like:

sql 
CREATE INDEX CONCURRENTLY idx_users_email ON users (email);

instead of blocking CREATE INDEX, when possible.

Docker and Container-Based Zero-Downtime

Containers make it easier to run multiple versions of your app at the same time, which is ideal for zero-downtime deployment.

Basic Pattern with Docker and Nginx

Imagine a simple setup:

Steps for a rolling update:

  1. Start containers with the new image:
bash 
   docker run -d --name app_v2_1 -p 8003:8000 myapp:2.0
   docker run -d --name app_v2_2 -p 8004:8000 myapp:2.0
  1. Add them to Nginx upstream:
nginx 
   upstream myapp {
       server 127.0.0.1:8001;  # old v1
       server 127.0.0.1:8002;  # old v1
       server 127.0.0.1:8003;  # new v2
       server 127.0.0.1:8004;  # new v2
   }
  1. Reload Nginx:
bash 
   nginx -s reload
  1. Monitor logs, metrics, and errors.
  2. When new version is stable, remove old instances:
    • Update upstream to only include ports 8003 and 8004.
    • Reload Nginx.
    • Stop old containers:
bash 
     docker stop app_v1_1 app_v1_2
     docker rm app_v1_1 app_v1_2

During this sequence, there is always at least one healthy instance of your app ready to serve traffic.

Docker Compose and Zero-Downtime

Docker Compose alone does not guarantee zero-downtime, but you can:

You can define multiple services for different versions, but in production you often move to more capable orchestrators, or manage Nginx manually.

Kubernetes and Rolling Updates

Kubernetes has built-in support for rolling updates and zero-downtime features:

A simplified flow in Kubernetes:

  1. You update the image in the Deployment spec.
  2. Kubernetes:
    • Creates a new pod with the new image.
    • Checks readiness probe.
    • When ready, sends traffic, then stops an old pod.
  3. It repeats until all old pods are replaced.

Kubernetes is beyond the scope of this chapter, but it is important to know that orchestration platforms can automate much of the zero-downtime process.

Handling Stateful Components

Zero-downtime is simpler when your application is stateless:

However, some components are stateful and need extra attention.

Sticky Sessions

Sticky sessions mean that a user is always routed to the same backend instance, usually to keep state in memory.

This is problematic for zero-downtime because:

Better approaches:

Background Jobs

Background workers and job queues are another stateful area:

Zero-downtime worker deployment:

  1. Stop sending new jobs to old workers:
    • For example, scale down old workers gradually.
  2. Let old workers finish their current tasks.
  3. Start new workers with the new code.
  4. Ensure job schemas and payload formats are backward compatible during overlap.

A common pattern: when you change the structure of job messages, follow an expand and contract pattern similar to database migrations.

Caches

Cached data can be:

To avoid cache-related downtime issues:

Step-by-Step Example: Rolling Update with FastAPI and Nginx

To make the ideas more concrete, here is a simplified step-by-step example that keeps downtime practically zero.

Assume:

1. Current State

Nginx configuration:

nginx 
upstream myapp {
    server 127.0.0.1:8001;  # app_v1_1
    server 127.0.0.1:8002;  # app_v1_2
}
server {
    listen 80;
    location / {
        proxy_pass http://myapp;
    }
}

Running containers:

bash 
docker run -d --name app_v1_1 -p 8001:8000 myapp:1.0
docker run -d --name app_v1_2 -p 8002:8000 myapp:1.0

2. Build and Start New Version

Build new Docker image:

bash 
docker build -t myapp:2.0 .

Start two new containers:

bash 
docker run -d --name app_v2_1 -p 8003:8000 myapp:2.0
docker run -d --name app_v2_2 -p 8004:8000 myapp:2.0

3. Update Load Balancer

Update upstream in Nginx:

nginx 
upstream myapp {
    server 127.0.0.1:8001;  # old v1
    server 127.0.0.1:8002;  # old v1
    server 127.0.0.1:8003;  # new v2
    server 127.0.0.1:8004;  # new v2
}

Reload Nginx gracefully:

bash 
nginx -s reload

At this point:

4. Monitor and Verify

Monitor:

If you detect serious problems, remove v2 instances from the config, reload Nginx, and stop v2 containers. All traffic returns to v1.

5. Remove Old Instances

When you are comfortable with v2:

  1. Update upstream to use only new instances:
nginx 
   upstream myapp {
       server 127.0.0.1:8003;  # new v2
       server 127.0.0.1:8004;  # new v2
   }
  1. Reload Nginx:
bash 
   nginx -s reload
  1. Stop and remove old containers:
bash 
   docker stop app_v1_1 app_v1_2
   docker rm app_v1_1 app_v1_2

During this entire process, Nginx always had at least one healthy backend, so users did not experience noticeable downtime.

Common Pitfalls and How to Avoid Them

Even with good plans, zero-downtime deployments can fail because of a few recurring mistakes.

Incompatible Changes Between Old and New Versions

Problem scenarios:

Avoid this by:

Ignoring Background Tasks and Scheduled Jobs

If you deploy only the web application but forget about workers:

Plan worker deployments as part of your overall zero-downtime strategy.

Lack of Monitoring and Observability

You cannot declare success for a deployment if you do not observe what happens.

At minimum, track:

More advanced metrics:

Zero-downtime is not only "the server stays up". It also means "users do not see new errors or degraded performance."

Not Testing the Deployment Process

You should:

You do not want the first time you run a new deployment approach to be in production.

Putting It All Together

Zero-downtime deployment combines several practices:

As your system grows, you might adopt sophisticated tools like Kubernetes, service meshes, and advanced CI/CD pipelines. However, the core ideas remain the same. Even with a basic setup of Nginx, Docker, and FastAPI, you can achieve near zero downtime by following the patterns described in this chapter.

Views: 8

Comments

Please login to add a comment.

Don't have an account? Register now!

© 2025 Kahibaro
Visitors: 3000 / 24h
Impressum / Imprint
Datenschutz / Data Policy