KAHIBARO
Discord Login Register ☰
☰

28.5. Fault Tolerance

Table of Contents

Why Fault Tolerance Matters

Fault tolerance is the ability of your backend system to continue working correctly when parts of it fail. In production, something is always failing: a node restarts, a network link drops, a disk becomes slow, a dependency has an outage.

Your goal is not to avoid every failure, which is impossible, but to design the system so that failures are contained, predictable, and recoverable.

Fault tolerance principle:
You will have failures. Design so that failures are expected, isolated, and handled instead of surprising and catastrophic.

In this chapter, the focus is on techniques and patterns that help your backend survive and recover from real-world problems, not on eliminating them.

Types of Failures in Production Systems

Before applying techniques, you need a vocabulary for failures. Different failures require different defenses.

Fail-stop vs Partial failures

Partial failures are often more dangerous because they are harder to detect and can cause cascading problems.

Transient vs Permanent failures

Fault-tolerant systems usually:

Local vs Cascading failures

Fault tolerance is largely about preventing local failures from becoming cascading failures.

Design Principles for Fault-Tolerant Systems

Prefer graceful degradation over total failure

A system that degrades is better than one that stops.

Examples of graceful degradation:

Try to answer:

Avoid single points of failure

A single point of failure (SPOF) is one component whose failure brings down the whole system.

Common SPOFs:

Approaches:

Design with failure as a normal path

In code, you often treat failures as "rare exceptions". In production, they are normal.

Practical implications:

A helpful mindset: every external call is "best effort", not guaranteed.

Redundancy and Replication

Redundancy is the core building block of fault tolerance: many weak parts can create a strong system.

Redundancy patterns

Horizontal redundancy

Run multiple instances of the same component.

Example:

Benefits:

Vertical redundancy

Increase the capacity or resources of a single machine:

Vertical redundancy alone does not solve fault tolerance, because the machine is still a single point of failure. Use it with horizontal redundancy, not instead of it.

Data replication

For stateful components like databases, simply adding more instances is not enough; their data must be replicated.

Common patterns:

Trade-offs:

Timeouts, Retries, and Backoff

External calls can fail or hang. Timeouts and retries are key tools to keep your application responsive and robust.

Timeouts

A timeout is the maximum time your code waits for an operation before giving up.

Why timeouts are critical:

Example in pseudocode:

python 
# Bad: no timeout
response = http_client.get("https://payments.example.com/pay")
# Better: explicit timeout
response = http_client.get("https://payments.example.com/pay", timeout=2.0)

Choose timeouts based on:

Rule: Every call to an external system must have a timeout.
No uncontrolled waiting for remote services.

Retries

Retries help with transient failures:

Do not blindly retry on every failure. Use selective retries:

Example retry logic (simplified):

python 
max_retries = 3
for attempt in range(max_retries):
    try:
        response = http_client.get(url, timeout=1.5)
        if response.status_code == 200:
            return response
        elif response.status_code in (502, 503, 504):
            # Retryable server errors
            continue
        else:
            break
    except NetworkError:
        # Retry on network errors
        continue
raise ExternalServiceError("Service unavailable after retries")

Exponential backoff

If many clients retry at the same time, they can amplify a problem and create a "retry storm".

Exponential backoff slows down retries after each failure, which gives the remote system time to recover.

Example backoff sequence:

Formula for backoff interval:

$$
t_n = t_0 \cdot 2^{(n - 1)}
$$

where:

Often, you also add jitter (randomness) to avoid synchronized retries.

Example:

python 
import random
import time
base_delay = 0.5
max_retries = 5
for attempt in range(1, max_retries + 1):
    try:
        return call_remote_service()
    except TransientError:
        delay = base_delay * (2 ** (attempt - 1))
        # Add jitter of +/- 50%
        jitter = random.uniform(0.5, 1.5)
        sleep_time = delay * jitter
        time.sleep(sleep_time)
raise ExternalServiceError("Failed after retries with backoff")

Circuit Breakers

A circuit breaker protects your system from repeatedly calling a failing dependency and making things worse.

It is inspired by electrical circuit breakers:

Circuit breaker states

A typical circuit breaker has three states:

StateDescriptionBehavior
ClosedEverything is normal.Allow all requests. Count failures.
OpenThe dependency is considered unhealthy.Immediately reject calls. No remote attempt.
Half-openTesting if the dependency has recovered.Allow a limited number of test calls.

Transition logic (simplified):

Why circuit breakers help

Without a circuit breaker:

With a circuit breaker:

Example behavior:

Bulkheads and Isolation

The bulkhead pattern separates parts of your system so that a failure in one area does not sink the entire "ship".

Resource isolation

Typical shared resources:

If one feature or dependency uses all of a shared resource, others get starved.

Bulkheads aim to:

Examples:

This way, if analytics queries go wild, they cannot consume all database connections and block the core business operations.

Example scenario

You have:

Without bulkheads:

With bulkheads:

Idempotency and Safe Retries

Idempotency is crucial for fault tolerance because retries can safely be applied only when operations are idempotent or at least safe to repeat.

What is idempotency?

An operation is idempotent if performing it multiple times has the same effect as performing it once.

Examples:

In HTTP:

Idempotency helps with:

Idempotency keys

For operations that are not naturally idempotent, you can make them idempotent with an idempotency key.

Typical pattern:

This is very useful for:

Idempotency does not eliminate all complexity, but it greatly simplifies safe retries and recovery.

Handling Partial Failures and Fallbacks

Some failures are local and can be worked around with fallback behavior.

Detecting partial failures

You can detect partial failures through:

Fallback strategies

Examples of fallbacks:

Practical question for each dependency:

Document the answers, and implement them in code.

Graceful Degradation and Feature Flags

Feature flags help you turn off risky parts quickly without redeploying.

Feature flags for fault tolerance

Use feature flags to:

Examples:

Feature flag systems can be:

Graceful degradation with flags

Combine feature flags with graceful degradation:

Self-Healing and Automated Recovery

Fault-tolerant systems often use self-healing mechanisms to detect and fix failures automatically.

Health checks and restarting

Many orchestrators and platforms, such as Kubernetes or Docker, support:

Patterns:

Auto-scaling and replacement

In cloud environments:

This does not replace good application behavior, but complements it.

Cleanup and compensation

Self-healing also includes:

Observability for Fault Tolerance

You cannot build fault-tolerant systems without understanding how they fail. Good observability is essential.

Key elements:

For fault tolerance, you should:

Trade-offs and Limitations

Every fault-tolerance technique has a cost. Overdoing them can hurt simplicity and performance.

Examples of trade-offs:

A useful guideline:

Summary

Fault tolerance is about expecting failure and planning for it. Core techniques include:

In production backend engineering, you do not aim for a system that never fails, but for one that fails predictably, recovers gracefully, and keeps the most important features working.

Views: 17

Comments

Please login to add a comment.

Don't have an account? Register now!

© 2025 Kahibaro
Visitors: 3086 / 24h
Impressum / Imprint
Datenschutz / Data Policy