KAHIBARO
Discord Login Register ☰
☰

28.6. Circuit Breakers

Table of Contents

Why Circuit Breakers Matter

In a production backend, failures are normal. Services can become slow, databases can overload, networks can be flaky. If your code keeps calling a failing dependency again and again, you waste resources and often make the situation worse.

A circuit breaker is a defensive pattern that stops your system from repeatedly calling something that is already known to be failing. It protects:

You can think of it as an automatic switch that cuts off a bad electrical circuit to prevent a fire. Here it cuts off bad remote calls to prevent cascading failures.

Key idea: A circuit breaker monitors calls to a dependency and opens when failures pass a threshold. While open, it blocks calls immediately, instead of trying and failing repeatedly.

Basic Circuit Breaker States

A circuit breaker has three main states that describe whether calls are allowed through.

Closed

In the Closed state, everything is normal.

If the failure rate crosses a configured threshold, the breaker opens.

Example configuration:

SettingExample value
Failure rate threshold50% failures
Minimum calls to assess20 calls
Sliding windowLast 30 seconds

If, within the last 30 seconds, at least 20 calls were made and 10 or more failed, the breaker moves from Closed to Open.

Open

In the Open state:

This prevents:

The breaker stays open for a configured cooldown period, for example 30 seconds. After that, it moves to Half-Open to test if the dependency has recovered.

Half-Open

In the Half-Open state:

Two outcomes are possible:

  1. Enough test calls succeed
    The dependency looks healthy again.
    The breaker moves to Closed, resets failure counters, and allows all calls again.
  2. Enough test calls fail
    The dependency is still unhealthy.
    The breaker moves back to Open and starts a new cooldown period.

Table summary:

StateCalls allowed?When entered
ClosedAll calls, until failures spikeOn startup or after Half-Open success
OpenNo calls (only fast fail/fallback)When failure threshold is exceeded in Closed
Half-OpenFew test calls allowedAfter Open timeout, to check if service recovered

What Circuit Breakers Protect Against

Circuit breakers are helpful in several common failure scenarios.

Slow or Hung Dependencies

Imagine your service calls a payment gateway that becomes very slow. Without a circuit breaker:

With a circuit breaker:

Cascading Failures

In a microservices system, one failing service can trigger failures in many others.

Without circuit breakers:

With circuit breakers between A and B, and between B and C:

Thundering Herd During Recovery

When a service starts to recover, many clients might retry at once and overload it again.

A Half-Open state helps here:

Key Configuration Parameters

You control the behavior of a circuit breaker by setting thresholds and timing values. Choosing good values is important.

Failure Thresholds

Two common types:

  1. Failure count threshold
    Open the breaker after a fixed number of failures in a row.

Example:

  1. Failure rate threshold
    Open the breaker when the ratio of failures to total calls is too high.

Example:

Typical rule:
Open the circuit when the failure rate exceeds a threshold (for example 50%) over a minimum number of recent calls (for example 20), within a sliding time window (for example 30 seconds).

Do not use very small windows, or your breaker might open and close too often because of random noise.

Timeouts and Durations

You usually configure:

If you pick a very short call timeout, you may mark a healthy but slow service as failing. If you pick a very long Open duration, recovery will be slow even when the dependency is fixed.

Allowed Calls in Half-Open

In Half-Open, you usually configure:

Example rule:

Implementing Circuit Breakers in Practice

You usually do not write a circuit breaker from scratch in production. Most languages have libraries that implement the pattern.

In Python, the simplest approach is a small custom implementation plus a decorator.

A Simple Python Circuit Breaker

This is a minimal example for education, not production ready. It shows the structure clearly.

python 
import time
from functools import wraps
class CircuitOpenError(Exception):
    pass
class CircuitBreaker:
    def __init__(
        self,
        failure_threshold=5,
        recovery_timeout=30,
        expected_exception=Exception,
    ):
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout
        self.expected_exception = expected_exception
        self.state = "closed"          # "closed", "open", "half_open"
        self.failure_count = 0
        self.last_failure_time = None
    def _open(self):
        self.state = "open"
        self.last_failure_time = time.time()
        self.failure_count = 0
    def _half_open(self):
        self.state = "half_open"
        self.failure_count = 0
    def _close(self):
        self.state = "closed"
        self.failure_count = 0
        self.last_failure_time = None
    def _can_attempt(self):
        if self.state == "closed":
            return True
        if self.state == "open":
            # Check if cooldown passed
            if (time.time() - self.last_failure_time) >= self.recovery_timeout:
                self._half_open()
                return True
            return False
        if self.state == "half_open":
            # Allow a single trial call at a time for simplicity
            return True
    def __call__(self, func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            if not self._can_attempt():
                raise CircuitOpenError("Circuit is open. Failing fast.")
            try:
                result = func(*args, **kwargs)
            except self.expected_exception:
                self.failure_count += 1
                if self.state == "half_open":
                    # Any failure in half-open re-opens
                    self._open()
                elif self.failure_count >= self.failure_threshold:
                    self._open()
                raise
            else:
                # Success
                if self.state in ("half_open", "open"):
                    self._close()
                else:
                    # Closed and successful, keep counters clean
                    self.failure_count = 0
                return result
        return wrapper

Usage example:

python 
import requests
breaker = CircuitBreaker(failure_threshold=3, recovery_timeout=10)
@breaker
def call_payment_service(order_id: str):
    response = requests.post(
        "https://payments.example.com/pay",
        json={"order_id": order_id},
        timeout=1.0,  # Network timeout, separate from breaker logic
    )
    response.raise_for_status()
    return response.json()
def pay_for_order(order_id: str):
    try:
        result = call_payment_service(order_id)
        return {"status": "ok", "payment_id": result["id"]}
    except CircuitOpenError:
        # Fallback: queue payment for later processing
        return {"status": "queued", "reason": "payment service unavailable"}
    except requests.RequestException:
        # Real failure
        return {"status": "failed", "reason": "payment request error"}

Flow:

Integrating with FastAPI

In FastAPI, you usually wrap client calls, not endpoints themselves. Example:

python 
from fastapi import FastAPI, HTTPException
import httpx
app = FastAPI()
weather_breaker = CircuitBreaker(failure_threshold=5, recovery_timeout=30)
@weather_breaker
def fetch_weather(city: str):
    with httpx.Client(timeout=1.0) as client:
        resp = client.get("https://weather.example.com/current", params={"city": city})
        resp.raise_for_status()
        return resp.json()
@app.get("/weather/{city}")
async def get_weather(city: str):
    try:
        data = fetch_weather(city)
        return {"city": city, "temperature": data["temp"]}
    except CircuitOpenError:
        # Return a clear but graceful error
        raise HTTPException(
            status_code=503,
            detail="Weather service temporarily unavailable. Please try again later.",
        )
    except httpx.HTTPError:
        # Other network or HTTP errors
        raise HTTPException(
            status_code=502,
            detail="Failed to fetch data from weather provider.",
        )

Here:

Fallback Strategies

When a circuit breaker opens, you have a choice: just error, or provide a fallback.

Common strategies:

StrategyDescriptionExample
Fail fastReturn an error immediately503 Service Unavailable
Cached responseReturn last known good dataCached product info
Degraded functionalityProvide a simpler version of the featureShow prices but disable live discounts
Queue for laterPut work into a background queue for retryQueue payments or emails
Alternative providerCall another similar service if one is downUse backup SMS or email provider

Example with a cache fallback:

python 
from fastapi import FastAPI, HTTPException
from typing import Dict
app = FastAPI()
product_cache: Dict[str, dict] = {}
product_breaker = CircuitBreaker(failure_threshold=3, recovery_timeout=20)
@product_breaker
def fetch_product_from_service(product_id: str) -> dict:
    # Imagine this calls a remote microservice
    ...
@app.get("/products/{product_id}")
def get_product(product_id: str):
    try:
        product = fetch_product_from_service(product_id)
        product_cache[product_id] = product  # Update cache
        return product
    except CircuitOpenError:
        # Try to serve from cache
        cached = product_cache.get(product_id)
        if cached:
            return {**cached, "warning": "Serving cached data"}
        raise HTTPException(
            status_code=503,
            detail="Product service unavailable and no cached data.",
        )

This provides a better user experience during outages, especially for mostly-read data like products or articles.

Circuit Breakers vs Retries

Circuit breakers and retries are often used together, but they serve different purposes and can conflict if used carelessly.

What Retries Do

A retry is when you try the same failing request again, usually with some delay. This can hide temporary network glitches.

Examples:

Dangerous Combination

If you blindly combine retries with circuit breakers:

To use both safely:

Important rule:
Never use unlimited or aggressive retries with circuit breakers. Retries should reduce noise from brief glitches, not hammer a failing service.

Where to Place Circuit Breakers

In a real system, you might have many dependencies.

Typical places to use circuit breakers:

You usually do not put a breaker around:

Instead of sprinkling circuit breaker code everywhere, it is better to:

Metrics, Logging, and Observability

Circuit breakers change how your system behaves under failure, so you must observe them.

Useful Metrics

Track at least:

If you use Prometheus, you might have metrics like:

Plot these in Grafana to see patterns like:

Logging

Log every state change:

These logs are very helpful during incident analysis, because they show when your system started protecting itself and when it recovered.

Common Mistakes and Pitfalls

Several problems appear often with circuit breakers.

Using One Global Breaker for Everything

If you use a single breaker for calls to many different services, a problem in one service will block all others.

Better:

Too Aggressive or Too Lazy Thresholds

You will typically need to adjust these values using real traffic data, not only guesses.

Ignoring Timeouts

A circuit breaker cannot fix calls that never time out. Always pair it with proper network timeouts and database timeouts.

Example:

python 
# Bad: no timeout, requests might hang for a long time
requests.get("https://service", params={"id": 1})
# Better: explicit timeout
requests.get("https://service", params={"id": 1}, timeout=1.0)

If you forget timeouts, failures may not be counted correctly, and your breaker logic will not trigger when it should.

Forgetting Idempotency

If you combine retries with circuit breakers, you will call the same dependency multiple times. For non-idempotent operations, this is risky.

Examples of non-idempotent operations:

In such cases:

Summary

Circuit breakers are a central part of building resilient backends in production:

In a production backend, you should:

Views: 7

Comments

Please login to add a comment.

Don't have an account? Register now!

© 2025 Kahibaro
Visitors: 3085 / 24h
Impressum / Imprint
Datenschutz / Data Policy