KAHIBARO
Discord Login Register ☰
☰

28.4. Graceful Shutdown

Table of Contents

Why Graceful Shutdown Matters

When a backend process stops, it can do so in two ways:

In production systems you almost always want graceful shutdown. It is important for:

A production backend must always implement a graceful shutdown procedure that:

  1. Stops accepting new work.
  2. Waits for in‑flight work to complete or times it out.
  3. Closes connections and external resources safely.
  4. Exits with a clear, predictable state.

Throughout this chapter we will look at patterns and concrete examples that you can adapt to your stack, with a focus on Python and async backends.

Signals and Process Lifecycle

On Unix-like systems (Linux, macOS) processes receive signals that tell them to stop or change behavior.

Common termination signals:

SignalTypical sourceMeaning in our context
SIGINTCtrl+C in a terminalInterrupt, usually used in development
SIGTERMsystemd, Kubernetes, Docker stopPlease terminate, used in production
SIGKILLKernel, kill -9Forced kill, cannot be caught

For graceful shutdown, we care about catchable signals, mainly SIGTERM and sometimes SIGINT.

High-level graceful shutdown flow:

  1. Process receives SIGTERM.
  2. A signal handler sets a flag, triggers cleanup logic, or stops servers.
  3. Servers stop accepting new connections.
  4. Ongoing requests get a chance to finish.
  5. Timeouts and resource cleanup run.
  6. Process exits with status code 0 (success).

If your process ignores SIGTERM, orchestration systems may escalate to SIGKILL, which gives you no chance to shut down gracefully.

Basic Pattern: Stop Accepting New Work

The first step of a graceful shutdown is to stop taking on new work.

For a web server, that means:

A common approach is to:

  1. On shutdown signal, close the listening socket or mark the server as shutting down.
  2. Allow existing connections to continue and finish.

In higher-level frameworks (like Uvicorn + FastAPI) this is often handled by the server. Your responsibility is to:

Tracking In‑Flight Requests

To avoid killing ongoing work, you need a way to know when it is safe to exit. A simple pattern is to track how many requests are currently running.

In pseudocode:

python 
in_flight_requests = 0
shutting_down = False
def before_request():
    global in_flight_requests
    if shutting_down:
        raise RejectNewRequests()
    in_flight_requests += 1
def after_request():
    global in_flight_requests
    in_flight_requests -= 1
def handle_shutdown():
    global shutting_down
    shutting_down = True
    wait_until(in_flight_requests == 0 or timeout_reached)

In an async Python service, you might do something similar with an asyncio.Lock or asyncio.Semaphore.

This pattern makes it possible to:

Timeouts and Maximum Shutdown Duration

You cannot wait forever. There will always be a request that hangs, a database that is slow, or a bug that delays completion.

So you usually combine:

Typical shutdown timeout values are 10 to 60 seconds, depending on your workload.

Always define a maximum shutdown timeout, for example:
$$ T_{\text{shutdown\_max}} = 30\ \text{seconds} $$
After $T_{\text{shutdown\_max}}$ expires, the process must exit even if work is still pending, otherwise deployments and scaling operations can hang.

In Kubernetes, for example, the terminationGracePeriodSeconds setting defines how long the platform will wait after sending SIGTERM before sending SIGKILL.

Graceful Shutdown in Async Python

Many modern backends use asyncio and async frameworks. In async systems, graceful shutdown is mostly about:

A typical pattern:

python 
import asyncio
import signal
shutdown_event = asyncio.Event()
def _signal_handler():
    shutdown_event.set()
async def main():
    loop = asyncio.get_running_loop()
    loop.add_signal_handler(signal.SIGTERM, _signal_handler)
    loop.add_signal_handler(signal.SIGINT, _signal_handler)
    # Start background tasks
    worker_task = asyncio.create_task(background_worker())
    # Wait for shutdown signal
    await shutdown_event.wait()
    # Begin graceful shutdown
    worker_task.cancel()
    try:
        await worker_task
    except asyncio.CancelledError:
        pass
    # Clean up resources here (DB, Redis, etc.)
asyncio.run(main())

Key ideas:

Coordinating with Web Servers (Uvicorn / Gunicorn / Nginx)

Graceful shutdown is not just your application. It is the whole chain:

  1. Load balancer / reverse proxy (Nginx, HAProxy, cloud LB).
  2. Application server (Gunicorn, Uvicorn workers).
  3. Application code (FastAPI, Django, etc.).
  4. Background workers (Celery, RQ, custom asyncio workers).

Each layer has a role.

Typical production flow:

  1. Load balancer marks the instance as draining so it does not receive new connections.
  2. Load balancer keeps existing connections alive for a grace period.
  3. System sends SIGTERM to the app server process.
  4. App server:
    • Stops accepting new connections.
    • Lets current requests finish.
    • Signals workers to shut down.
  5. Your app:
    • Releases resources.
    • Stops background tasks.

You need to ensure your application hooks run, for example:

Handling Database Connections and Pools

Databases maintain connections that are expensive to create. You usually use a connection pool.

During graceful shutdown, you must:

If you use an ORM, it likely provides a method such as:

Example pattern (pseudocode):

python 
async def on_shutdown():
    # Stop accepting new requests, wait for ongoing ones ...
    # Close DB connection pool
    await async_engine.dispose()

If you do not close the pool, the OS will eventually free the resources when the process exits, but some drivers or libraries may not flush buffered data or metrics correctly.

Also ensure that long-running transactions are avoided in your design, because they make graceful shutdown harder. Use shorter transactions and commit frequently.

Shutting Down Background Workers and Queues

Web requests are not the only form of work. You may have:

For each of these:

  1. On shutdown signal, ask the worker to stop fetching new jobs.
  2. Let the current job finish (or interrupt it in a safe way).
  3. Acknowledge or requeue jobs appropriately.
  4. Close connections to the broker.

Example pattern with an async worker loop:

python 
async def worker(shutdown_event):
    while not shutdown_event.is_set():
        job = await queue.get()
        try:
            await handle_job(job)
        finally:
            queue.task_done()

When shutdown_event is set:

In job systems like Celery, built‑in signals and commands handle much of this, but you still need to design your tasks to be idempotent and able to resume if interrupted.

Draining Traffic Behind a Load Balancer

In a production environment, your backend is usually behind a load balancer or ingress controller.

You typically combine two mechanisms:

  1. Instance draining at the load balancer:
    • Stop routing new requests to a particular instance.
    • Keep existing connections alive for a configured delay.
  2. Process-level graceful shutdown:
    • The instance still serves the connections it has.
    • It waits for those requests to finish.

Typical Kubernetes scenario:

  1. Pod is marked for termination.
  2. Kubernetes sends SIGTERM to containers.
  3. At the same time, the pod is removed from the service endpoints, so the service stops sending new requests.
  4. Pod has up to terminationGracePeriodSeconds to finish ongoing work and exit.

Similar concepts exist in cloud load balancers, like ALB/NLB in AWS, where you configure:

Coordinating these settings with your app’s shutdown timeout is important. If the load balancer stops routing traffic too late, you may still receive new requests already in your shutdown phase.

Handling Long‑Running and Streaming Requests

Some endpoints may:

For such endpoints you must decide:

Options include:

It is usually a bad idea to keep a process alive indefinitely just because one client has kept a streaming connection open. Use a reasonable time limit for such connections.

Observability During Shutdown

To understand and trust your shutdown behavior, you should:

This helps you answer questions like:

A Typical Graceful Shutdown Checklist

Use this as a practical reference when implementing graceful shutdown in a backend service.

Graceful Shutdown Checklist

  1. Signal handling
    • Handle SIGTERM (and optionally SIGINT).
    • Trigger a shutdown event instead of exiting immediately.
  2. Stop accepting new work
    • Stop accepting new web requests.
    • Stop fetching new jobs from message queues.
  3. Track in‑flight work
    • Maintain a count or registry of active requests / jobs.
    • Wait for them to finish, subject to a maximum shutdown timeout.
  4. Timeouts
    • Set per‑request or per‑job timeouts.
    • Set a global shutdown timeout $T_{\text{shutdown\_max}}$.
  5. Release resources
    • Close DB connections and pools.
    • Close Redis, message broker, file handles, and external client connections.
  6. Cooperate with load balancers
    • Ensure instance draining is configured.
    • Coordinate LB grace period with your shutdown timeout.
  7. Logs and metrics
    • Log shutdown steps and durations.
    • Expose metrics related to shutdown success / failures.
  8. Test it
    • Simulate SIGTERM in staging or local environment.
    • Verify that:
      • No new requests are accepted.
      • Ongoing requests finish successfully.
      • The process exits within the configured time.

By following these patterns, you can make your backend behave predictably under restarts, deployments, and failures, which is essential for reliable production systems.

Views: 8

Comments

Please login to add a comment.

Don't have an account? Register now!

© 2025 Kahibaro
Visitors: 3504 / 24h
Impressum / Imprint
Datenschutz / Data Policy