KAHIBARO
Discord Login Register ☰
☰

17.7. Retry Strategies

Table of Contents

Why Retry Strategies Matter

In background processing, things often fail temporarily. A database might be momentarily overloaded, an email provider might return a 500 error, or a network link might drop for a second.

If you treat every error as permanent, you will lose work. If you treat every error as temporary and retry blindly, you can overload your own systems or external services.

A retry strategy is a set of rules that decide:

A good strategy maximizes successful completion of tasks while avoiding overload and infinite loops.

A background system without a well‑defined retry strategy is a reliability risk. It can either lose important work or create retry storms that overload your own services and external APIs.

What Can Be Safely Retried?

Not every failure should be retried. Some are permanent and will never succeed, others are transient and might succeed later.

Transient vs Permanent Errors

Examples of transient errors (good candidates for retry):

Examples of permanent errors (usually do not retry):

In code, you usually:

Example in Python‑style pseudocode:

python 
class PermanentError(Exception):
    pass
class TransientError(Exception):
    pass
def process_job(job):
    if not job.data_is_valid():
        raise PermanentError("Invalid job data")
    try:
        external_service_call(job)
    except TimeoutError as exc:
        raise TransientError("Temporary network issue") from exc

The worker’s retry logic will treat PermanentError and TransientError very differently.

Basic Retry Parameters

Every retry strategy is defined by a few key parameters.

Maximum Number of Attempts

You almost always define a maximum number of attempts, including the first try.

For example:

Never allow unlimited retries for a job. Always set a maximum number of attempts or a maximum total time window for retries.

Delay Between Attempts

You also define how long to wait before each retry:

Even a small delay reduces pressure on failing systems.

Fixed Delay Strategy

The simplest strategy is fixed delay.

Example:

Timeline:

Attempt | Result | Next retry in
--------|---------|---------------
1 | Fails | 10 s
2 | Fails | 10 s
3 | Fails | 10 s
4 | Fails | 10 s
5 | Fails | Stop

Pseudocode:

python 
max_attempts = 5
delay = 10  # seconds
for attempt in range(1, max_attempts + 1):
    try:
        perform_task()
        break
    except TransientError as exc:
        if attempt == max_attempts:
            log_failure(exc)
            break
        sleep(delay)

When fixed delay is useful:

Downside:

Exponential Backoff

Exponential backoff increases the delay after each failure. Each retry waits longer than the previous one, typically doubling the wait time.

Common formula:

$$
\text{delay}_n = \text{base\_delay} \times 2^{(n - 1)}
$$

Where:

Example with base_delay = 2 seconds, max_attempts = 5:

Attempt | Delay before this attempt | Comment
--------|---------------------------|--------
1 | 0 s | First attempt
2 | $2 \times 2^{0} = 2$ s | 2 seconds
3 | $2 \times 2^{1} = 4$ s | 4 seconds
4 | $2 \times 2^{2} = 8$ s | 8 seconds
5 | $2 \times 2^{3} = 16$ s | 16 seconds

Total waited time is $2 + 4 + 8 + 16 = 30$ seconds.

Pseudocode:

python 
max_attempts = 5
base_delay = 2  # seconds
attempt = 1
while attempt <= max_attempts:
    try:
        perform_task()
        break
    except TransientError as exc:
        if attempt == max_attempts:
            log_failure(exc)
            break
        delay = base_delay * (2 ** (attempt - 1))
        sleep(delay)
        attempt += 1

Benefits:

Exponential Backoff With Maximum Delay

Pure exponential backoff can grow too large. You usually add a maximum delay cap.

Always set a maximum backoff delay to avoid extremely long waits. For example, cap delays at 1 minute or 5 minutes.

Formula with cap:

$$
\text{delay}_n = \min\left(\text{base\_delay} \times 2^{(n - 1)}, \text{max\_delay}\right)
$$

Example:

Attempt | Raw delay | Capped delay
--------|-----------|-------------
1 | 0 | 0
2 | 2 | 2
3 | 4 | 4
4 | 8 | 8
5 | 16 | 16
6 | 32 | 30 (capped)
7 | 64 | 30 (capped)

Jitter: Avoiding Thundering Herds

If all workers use the same retry schedule, they may all retry at exactly the same times. This can cause a thundering herd effect: many clients hit a recovering service at once and keep it down.

Jitter is randomization added to the delay. Instead of sleeping exactly 10 seconds, you sleep a random duration around 10 seconds.

Common patterns:

A simple full jitter version:

$$
\text{delay}_n = \text{random}(0, \text{base\_delay} \times 2^{(n - 1)})
$$

Example with full jitter:

Pseudocode with full jitter:

python 
import random
import time
max_attempts = 5
base_delay = 2  # seconds
max_delay = 30  # seconds
for attempt in range(1, max_attempts + 1):
    try:
        perform_task()
        break
    except TransientError as exc:
        if attempt == max_attempts:
            log_failure(exc)
            break
        # Exponential backoff with cap
        backoff = base_delay * (2 ** (attempt - 1))
        backoff = min(backoff, max_delay)
        # Apply jitter
        delay = random.uniform(0, backoff)
        time.sleep(delay)

Large systems often use some form of exponential backoff with jitter for external calls.

Time‑Based vs Attempt‑Based Limits

You can limit retries by:

Example combined rule:

Pseudocode:

python 
MAX_ATTEMPTS = 10
MAX_AGE_SECONDS = 3600
def should_retry(job, attempt, error):
    if attempt >= MAX_ATTEMPTS:
        return False
    if time.time() - job.created_at_timestamp > MAX_AGE_SECONDS:
        return False
    if isinstance(error, PermanentError):
        return False
    return True

This prevents stuck jobs that keep retrying for days.

Idempotency and Retries

When you retry a job, the code can run multiple times for the same logical operation. You must make sure retries do not create unwanted side effects.

An operation is idempotent if performing it multiple times has the same effect as performing it once.

Examples:

For background jobs:

To make non‑idempotent operations safer, you can:

Example: avoid double charging with an idempotency key:

python 
def charge_user(user_id, amount, operation_id):
    if payment_already_processed(operation_id):
        return  # Do nothing, this retry is safe
    # perform actual charge
    result = payment_gateway.charge(user_id, amount)
    # store operation_id as processed
    mark_payment_processed(operation_id, result)

If the job retries with the same operation_id, it will not charge again.

Never implement retries for non‑idempotent operations without an explicit strategy to avoid duplicate side effects.

Categorizing Failures for Retry

In a real system you often decide whether to retry based on error types, error codes, or HTTP status codes.

HTTP Based Decisions

Typical rules for calling external HTTP APIs:

Status | Retry? | Reason
-------|--------|-------
2xx | No | Request succeeded
4xx | No | Client or input error, fix the request instead
408 | Yes | Request timeout, usually transient
429 | Yes | Too many requests, back off and retry later
5xx | Yes | Server errors are usually transient
599 | Yes | Network timeout or proxy error

Example:

python 
def should_retry_http(status_code: int) -> bool:
    if 200 <= status_code < 300:
        return False
    if status_code == 408:
        return True
    if status_code == 429:
        return True
    if 500 <= status_code < 600:
        return True
    return False

Database Based Decisions

For databases, you can retry some errors:

Retryable:

Not retryable:

You usually map database error codes to retryable / non‑retryable classifications.

Retry Strategies in Job Queues

Most background job systems and message queues have built‑in support for retries. You configure the strategy instead of writing all logic yourself.

Example: Simple Retry in a Worker

A generic worker pattern:

python 
def worker_loop():
    while True:
        job = queue.pop()
        attempt = job.attempts + 1
        try:
            process_job(job)
            mark_done(job)
        except TransientError as exc:
            reschedule(job, attempt, exc)
        except PermanentError as exc:
            mark_failed(job, exc)

Where reschedule uses your strategy:

python 
def reschedule(job, attempt, error):
    delay = compute_delay(attempt)
    if not should_retry(job, attempt, error):
        mark_failed(job, error)
        return
    job.attempts = attempt
    job.next_run_at = now() + delay
    queue.push(job)

Example Settings for a Real System

Here is a realistic configuration you might choose for sending emails:

Parameter | Value
------------------|-------
Max attempts | 8
Base delay | 5 seconds
Max delay | 15 minutes
Backoff | Exponential with jitter
Total time limit | 24 hours
Retry only on | Network errors and provider 5xx codes

For payment jobs you might choose something stricter:

Parameter | Value
------------------|-------
Max attempts | 4
Base delay | 10 seconds
Max delay | 5 minutes
Backoff | Exponential with jitter
Total time limit | 1 hour
Retry only on | Explicitly marked retryable errors from gateway
Idempotency key | Required per operation

Guardrails and Monitoring

Retries can hide problems. If every third attempt fails but retries succeed, users might not notice, but your system is under stress.

You need metrics and alerts:

Example alerts:

Also consider circuit breakers (covered elsewhere in the course). If too many retries fail, you may temporarily stop sending new requests to a service, instead of endlessly retrying.

Practical Guidelines

To design a retry strategy for a new background task:

  1. Decide if the task is safe to retry.
    • If not idempotent, design idempotency first.
  2. Identify retryable errors.
    • Map exceptions and status codes to retryable / non‑retryable.
  3. Choose limits.
    • max_attempts and possibly a total time limit.
  4. Pick a delay pattern.
    • For most cases, use exponential backoff with jitter and a max delay.
  5. Set clear failure behavior.
    • Where do jobs go after final failure, for example a dead‑letter queue?
  6. Monitor and adjust.
    • Start conservative, then tune based on real failure patterns.

If you apply these rules consistently, your background processing system will be both robust and gentle on the systems it depends on.

Views: 18

Comments

Please login to add a comment.

Don't have an account? Register now!

© 2025 Kahibaro
Visitors: 3467 / 24h
Impressum / Imprint
Datenschutz / Data Policy