KAHIBARO
Discord Login Register ☰
☰

17.8. Handling Failed Jobs

Table of Contents

Why Failed Jobs Matter

Background jobs are supposed to run without user interaction, often after a request is done. When they fail, the user might not see an error directly, but important work is not done. For example:

If you do not handle failed jobs, you will get:

Good failure handling makes your system more reliable and easier to operate.

Key rule:
Every background processing system must have a clear strategy for detecting, recording, retrying, and sometimes manually fixing failed jobs.

Types of Failures in Background Jobs

Temporary vs Permanent Failures

You should separate failures into two categories:

TypeDescriptionExamplesTypical Action
TemporaryLikely to succeed if tried againNetwork glitches, rate limits, timeoutsRetry later
PermanentWill not succeed without a changeInvalid email, deleted user, bad configDo not retry blindly

Examples:

If you treat permanent failures like temporary ones, you might create an infinite retry loop and overload your system.

Synchronous vs Asynchronous Failures

Failures can happen:

You mainly focus on failures in the worker, but do not forget the others. For example, if enqueueing fails, you may need to:

Detecting and Recording Failed Jobs

Logging Failures

Every failed job must be logged with enough context:

Example in Python with Celery-style code:

python 
import logging
logger = logging.getLogger(__name__)
def send_welcome_email(user_id: int):
    try:
        user = get_user_from_db(user_id)
        email_service.send_welcome_email(user.email)
    except Exception as exc:
        logger.exception("Failed to send welcome email", extra={"user_id": user_id})
        raise

logger.exception automatically adds the stack trace.

Storing Failed Job Metadata

Log files are not enough. You often want a structured record, for example in:

Example of a failed_jobs table:

ColumnTypeDescription
idUUIDUnique identifier
job_typetextName of the job function
payloadjsonbJob arguments
error_messagetextShort error description
error_typetextException class name
stack_tracetextOptional, for debugging
retriesintHow many times tried
statustexte.g. failed, will_retry
created_attimestampWhen first failed
last_attempt_attimestampWhen last retried

This lets you:

Dead-Letter Queues (DLQ)

A dead-letter queue is a special queue where messages (jobs) go when they cannot be processed successfully.

Basic idea:

  1. Worker receives job from main queue.
  2. Worker tries to process it up to N times.
  3. If it still fails, job is moved to the DLQ instead of being lost.

Benefits:

Many message brokers support DLQ behavior, for example:

Retry Strategies

Simple Retries

The most basic approach is to retry a job a fixed number of times.

Example logic:

Pseudo code:

python 
MAX_RETRIES = 3
DELAY_SECONDS = 10
def process_job(job):
    for attempt in range(1, MAX_RETRIES + 1):
        try:
            do_work(job)
            return  # success
        except Exception as exc:
            if attempt == MAX_RETRIES:
                move_to_dead_letter(job, exc)
            else:
                sleep(DELAY_SECONDS)

Problems with fixed delay:

Exponential Backoff

Exponential backoff means that waiting time increases with each retry. This is common in resilient systems.

Formula example:

If base_delay = 5 seconds:

AttemptDelay (seconds)
15
210
320
440

Important rule:
Use exponential backoff for retries of external calls like APIs or email servers to avoid overloading them and to give them time to recover.

Example:

python 
import time
MAX_RETRIES = 5
BASE_DELAY = 5  # seconds
def send_notification_with_retry(notification):
    for attempt in range(1, MAX_RETRIES + 1):
        try:
            send_notification(notification)
            return
        except TemporaryError as exc:
            if attempt == MAX_RETRIES:
                raise
            delay = BASE_DELAY * (2 ** (attempt - 1))
            time.sleep(delay)

Adding Jitter

If many workers retry at the same time with the same pattern, you can create "retry storms". To avoid this, add randomness (jitter) to the delay.

Example:

python 
import random
import time
def compute_delay_with_jitter(base_delay: int, attempt: int) -> float:
    delay = base_delay * (2 ** (attempt - 1))
    jitter = random.uniform(0, delay * 0.1)  # up to 10% extra
    return delay + jitter

This spreads retries over time and smooths the load.

Retry Only Safe Operations

Some jobs are safe to retry, some are not.

Safe (idempotent or easy to detect duplicates):

Dangerous:

For non-idempotent jobs, use:

Example: payment job that uses an idempotency key and is safe to retry.

python 
def charge_customer(order_id: int, idempotency_key: str):
    if payment_exists(idempotency_key):
        return  # already processed
    result = payment_gateway.charge(order_id, idempotency_key=idempotency_key)
    save_payment(result, idempotency_key)

Distinguishing Retryable vs Non-Retryable Errors

Classifying Exceptions

Define which errors can be retried and which cannot.

Example categories:

In Python you can create custom exception classes:

python 
class RetryableError(Exception):
    pass
class NonRetryableError(Exception):
    pass

Use them in your job:

python 
def send_invoice(order_id: int):
    order = get_order(order_id)
    if not order:
        raise NonRetryableError(f"Order {order_id} not found")
    try:
        pdf = generate_invoice_pdf(order)
    except PdfEngineTimeout as exc:
        raise RetryableError("PDF service timeout") from exc
    email_service.send_attachment(order.customer_email, pdf)

Your worker can then decide:

python 
try:
    send_invoice(order_id)
except RetryableError as exc:
    schedule_retry(order_id, error=exc)
except NonRetryableError as exc:
    record_permanent_failure(order_id, error=exc)

Blacklist vs Whitelist Approach

Two strategies to decide what to retry:

For beginners, whitelist is often safer, for example:

Limiting Retries and Preventing Infinite Loops

Maximum Attempts

Each job should have a clear limit:

Store the attempt count as:

Example:

python 
MAX_ATTEMPTS = 5
def should_retry(job):
    return job.attempts < MAX_ATTEMPTS

If you skip this, you may create:

Circuit Breaker Concepts

A circuit breaker stops making calls to a failing dependency for a while.

In the context of jobs, if you see many failures for a specific dependency:

This protects:

Full circuit breaker patterns are an advanced topic, but as a beginner you should at least:

Manual Intervention and Admin Tools

Inspecting Failed Jobs

You need tools to view failed jobs, for example:

Useful information to show:

Example of a simple view in a web admin:

IDJob TypeError MessageAttemptsLast Attempt At
1send_welcomeSMTP timeout32026-08-28 13:05:21
2generate_reportUser 42 not found12026-08-28 13:07:10

Requeueing Jobs

After you fix a bug or a configuration, you may want to re-run some failed jobs.

Possible actions from your admin tool:

When requeueing:

Pseudo code for requeueing a job:

python 
def requeue_failed_job(failed_job_id: int):
    failed_job = db.get_failed_job(failed_job_id)
    enqueue_job(
        type=failed_job.job_type,
        payload=failed_job.payload,
        attempts=failed_job.retries + 1
    )
    db.mark_failed_job_as_requeued(failed_job_id)

Marking Jobs as Resolved

Not every failed job should be re-run. Sometimes you decide:

In such cases, mark the job as "resolved" without retry.

This prevents your failure lists from growing forever and lets you track what is still pending.

Idempotency and Safe Re-Runs

What Is Idempotency in Jobs?

A job is idempotent if running it multiple times produces the same result as running it once.

Examples:

Idempotency is very helpful for failed jobs because you can:

Techniques for Idempotent Jobs

Some practical methods:

  1. Use unique constraints in DB

Example: a payments table with unique constraint on order_id:

sql 
   ALTER TABLE payments
   ADD CONSTRAINT unique_order_payment UNIQUE (order_id);

If a job tries to insert a second payment for the same order, it fails, and you can catch that as "already processed".

  1. Use idempotency keys

You store a unique key for each logical operation. If you see the same key again, you return the previous result instead of doing the work again.

  1. Check state before acting

For example:

python 
   def mark_order_as_shipped(order_id: int):
       order = get_order(order_id)
       if order.status == "shipped":
           return  # already shipped
       update_order_status(order_id, "shipped")

Running this twice does not break anything.

Observability for Failed Jobs

Metrics

Collect metrics about your job system, such as:

You can then:

Example metric names:

Metric NameMeaning
jobs_processed_totalTotal number of processed jobs
jobs_failed_totalTotal number of failed jobs
jobs_retry_totalTotal number of retries
job_duration_secondsExecution time per job

Traces and Correlation IDs

Correlate jobs with user actions using:

This makes debugging much easier. For example:

Then you can trace the full flow through system logs.

Putting It All Together: Example Flow

Consider a job send_order_confirmation_email(order_id):

  1. API receives a request to place an order.
  2. After saving the order, API enqueues a job with order_id and request_id.
  3. Worker picks the job and runs it.

Possible outcomes:

This flow avoids losing important work, keeps the system stable, and gives you tools to recover from problems.

Views: 17

Comments

Please login to add a comment.

Don't have an account? Register now!

© 2025 Kahibaro
Visitors: 3044 / 24h
Impressum / Imprint
Datenschutz / Data Policy