KAHIBARO
Discord Login Register ☰
☰

17.2. Background Jobs

Table of Contents

Why Background Jobs Exist

When you build a backend, many tasks do not need to finish while the user is waiting for a response. These are perfect candidates for background jobs.

Examples of such tasks:

If you do these inside the main HTTP request, the user waits. If the email provider is slow or a report takes 20 seconds to generate, the user sees a spinning loader or a timeout. With background jobs, the API can:

  1. Accept the request.
  2. Save what is needed to process the task later.
  3. Respond quickly.
  4. Process the heavy work in the background.

Key rule: Any work that is slow, not user-facing, or can be retried later is usually a good candidate for a background job.

Synchronous Work vs Background Jobs

To understand background jobs, contrast them with synchronous work.

Synchronous work in an HTTP handler

Synchronous means your application waits until the task is done.

Example without background jobs:

python 
@app.post("/register")
def register_user(user: UserIn):
    # 1. Create user in database
    db_user = create_user(user)
    # 2. Send welcome email (might be slow)
    send_welcome_email(db_user.email)
    # 3. Return response
    return {"id": db_user.id, "email": db_user.email}

If send_welcome_email takes 5 seconds, the user waits 5 seconds.

Problems:

Moving work to a background job

With a background job, you split the flow:

python 
@app.post("/register")
def register_user(user: UserIn):
    db_user = create_user(user)
    # Enqueue background job
    task_id = enqueue_send_welcome_email(db_user.id)
    return {
        "id": db_user.id,
        "email": db_user.email,
        "welcome_email_task_id": task_id,
    }

A separate worker process later runs:

python 
def process_send_welcome_email(user_id: int):
    user = get_user_by_id(user_id)
    send_welcome_email(user.email)

The HTTP request is now fast. Email processing is independent, can be retried, and can be monitored.

Types of Background Jobs

Not all background jobs are the same. It is useful to classify them, because each type has slightly different requirements.

1. Immediate jobs

These should run as soon as possible after the request, but not necessarily inside the request.

Examples:

Typical behavior:

Pseudocode example:

python 
# In your API
task_id = job_queue.enqueue("send_confirmation_email", user_id=user.id)
# In your worker
def send_confirmation_email(user_id):
    # send the email here
    pass

2. Delayed jobs

These should run later, not immediately.

Examples:

Typical behavior:

Pseudocode:

python 
# run in 24 hours
job_queue.enqueue_in(
    delay_seconds=24 * 3600,
    task_name="send_event_reminder",
    event_id=event.id,
)

3. Periodic or scheduled jobs

These run repeatedly on a schedule.

Examples:

This is similar to a cron job. Some job systems integrate scheduling, but often you use a separate scheduler that enqueues jobs.

Example schedules:

ScheduleExample task
Every minutePoll an external service
Every 5 minutesRebuild search index chunk
Every hourGenerate aggregate metrics
Every day at 01:00Delete old temporary uploads

4. Long-running jobs

These take a long time, for example several minutes or more.

Examples:

These jobs often need:

Common Use Cases with Examples

Here are concrete scenarios that are very common in backend systems.

Sending emails

Sending email is a classic background job.

Basic flow:

  1. API receives a request that needs an email.
  2. API enqueues a job like send_email(user_id, template_name).
  3. Worker pulls the job, loads the user, renders the email, sends it.

Example using pseudocode:

python 
# API handler
def post_register(user):
    new_user = create_user(user)
    job_queue.enqueue("send_welcome_email", user_id=new_user.id)
    return {"id": new_user.id}
# worker
def send_welcome_email(user_id):
    user = get_user(user_id)
    body = render_template("welcome_email.html", user=user)
    email_client.send(to=user.email, subject="Welcome!", body=body)

Image processing

Image-related operations are often slow and CPU intensive.

Tasks:

Example:

python 
# API: upload endpoint
def upload_image(file):
    image_id = save_image_original(file)
    job_queue.enqueue("generate_thumbnails", image_id=image_id)
    return {"image_id": image_id}
# worker
def generate_thumbnails(image_id):
    original = load_image(image_id)
    for size in [64, 128, 256]:
        thumbnail = resize_image(original, size)
        save_thumbnail(image_id, size, thumbnail)

Data imports and exports

Big data files, like CSV imports, should not be processed in a synchronous request.

Flow:

  1. User uploads a CSV file.
  2. API stores file and enqueues job import_csv(file_path, user_id).
  3. Worker reads the file, validates rows, inserts data into DB.
  4. Worker updates some import_status table with progress.

Example of tracking import:

ColumnDescription
idImport job id
user_idWho requested the import
statuspending, processing, completed, failed
total_rowsTotal rows detected
processedRows already processed
errorsError message or count

The frontend can poll /imports/{id} to show progress.

Cleanup and maintenance

Background jobs are great for maintenance tasks.

Examples:

Scheduled job:

python 
# runs every night
def cleanup_old_sessions():
    delete_from_sessions_where_expires_before(now())

Designing Background Jobs

Good background job design makes your system reliable and easier to debug.

Job payloads: what to put into a job

A job has a payload, the data needed to run it. Design payloads carefully.

Better to pass:

Avoid passing:

Example of a good payload:

json 
{
  "task": "send_order_confirmation",
  "order_id": 12345
}

The worker can load the order details from the database. If order details change later, the job uses the most up-to-date information.

Idempotency: safe to run multiple times

An idempotent job can run once, twice, or many times with the same result.

Important rule: Design background jobs to be idempotent so retries and duplicates do not cause data corruption or duplicate side effects.

Examples:

python 
def send_welcome_email(user_id):
    user = get_user(user_id)
    if user.welcome_email_sent:
        return  # already sent, do nothing
    actually_send_welcome_email(user.email)
    mark_welcome_email_sent(user_id)

Job naming and organization

Group jobs logically:

This helps with:

You can also place jobs in modules or packages that reflect your domain.

Background Job Lifecycle

Almost every job follows a simple lifecycle:

  1. Enqueued
    The application creates a job and puts it into a queue.
  2. Picked up by a worker
    A worker process retrieves a job from the queue.
  3. Processing
    The job handler runs your code.
  4. Completed
    If it succeeds, the job is marked completed and removed from the queue, or its result is stored.
  5. Failed
    If there is an error, the job may be retried or marked failed permanently.

You might also track extra states:

Example state transitions:

FromToReason
queuedprocessingWorker starts work
processingcompletedJob finished without error
processingfailedJob raised an exception
failedqueuedRetry scheduled
queuedscheduledIf it is a job for a future time

Error Handling and Retries

Background jobs will fail sometimes. Maybe:

You cannot avoid failures, but you can design how to respond.

Automatic retries

Common strategy:

A simple retry policy:

AttemptDelay before next attempt
10 seconds (first)
210 seconds
360 seconds
4300 seconds

This is an example of exponential backoff.

Key rule: Always use retries with backoff for operations that can fail due to temporary issues like network problems.

Distinguish between permanent and temporary errors

Some errors are permanent, retried attempts will never succeed:

Some errors are temporary:

Your job code can:

Example:

python 
def send_email(user_id):
    user = get_user(user_id)
    if not is_valid_email(user.email):
        # permanent failure
        raise PermanentJobError("Invalid email")
    try:
        email_client.send(user.email, ...)
    except NetworkError as e:
        # temporary, should be retried
        raise TemporaryJobError(str(e))

The job system can use error type to decide retry behavior.

Dead-letter queues

A dead-letter queue (DLQ) holds jobs that failed permanently after all retries.

Why keep them:

Flow:

  1. Job fails many times.
  2. Job is moved to the DLQ with its last error.
  3. A developer or admin reviews DLQ periodically.

Monitoring and Observability for Jobs

Background jobs are invisible to users, so you must add ways to observe them.

Basic metrics

Useful metrics for job systems:

MetricMeaning
Queue lengthHow many jobs are waiting
Jobs processed per minuteThroughput
Average job duration by typeHow long a job takes
Number of failures per typeWhich job types are most problematic
Retry countsHow many retries are happening

If the queue grows constantly, workers may be too few or jobs too slow.

Logs

You should log:

Example log structure:

json 
{
  "event": "job_started",
  "job_id": "abc123",
  "job_type": "send_welcome_email",
  "timestamp": "2026-08-27T12:00:00Z"
}
json 
{
  "event": "job_failed",
  "job_id": "abc123",
  "job_type": "send_welcome_email",
  "error": "Network timeout",
  "attempt": 2,
  "timestamp": "2026-08-27T12:00:10Z"
}

Dashboards

For serious systems, you want:

Many job processing tools provide this out of the box, or you can build simple endpoints that query job status and show it.

Patterns and Best Practices

Here are patterns that help you design reliable background jobs.

1. Keep jobs small and focused

One job should do one clear thing. For example:

Small jobs:

If you need several steps, you can chain jobs:

  1. Job generate_invoice once done, enqueues
  2. Job send_invoice_email, which then enqueues
  3. Job update_invoice_status.

2. Do not block jobs on user input

Jobs should run automatically. They should not ask for more input from the user. If a job needs parameters, the request that enqueues it must provide them.

3. Store any important results

If a job produces a result that you need later, store it in a database or file storage.

Examples:

A job should not just log the result; logs are not designed for easy querying.

4. Avoid heavy shared state

Jobs are often processed by multiple workers, possibly on different machines.

Avoid:

Use:

5. Consider ordering requirements

Sometimes order matters. For example, you might need:

You can handle this in multiple ways:

Simple Background Job Implementation Idea

Even without a full job framework, you can build a very simple job system for learning purposes.

Example: Database-backed job queue

Table:

sql 
CREATE TABLE jobs (
    id SERIAL PRIMARY KEY,
    type TEXT NOT NULL,
    payload JSONB NOT NULL,
    status TEXT NOT NULL DEFAULT 'queued',
    attempts INT NOT NULL DEFAULT 0,
    last_error TEXT,
    created_at TIMESTAMP NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP NOT NULL DEFAULT NOW()
);

Workflow:

  1. API inserts a job:
sql 
INSERT INTO jobs (type, payload) VALUES ('send_email', '{"user_id": 1}');
  1. Worker loop (pseudocode):
python 
while True:
    job = fetch_next_queued_job()
    if not job:
        sleep(1)
        continue
    try:
        mark_job_as_processing(job.id)
        run_job(job.type, job.payload)
        mark_job_as_completed(job.id)
    except Exception as e:
        increment_attempts_and_record_error(job.id, str(e))
        if job.attempts >= MAX_ATTEMPTS:
            mark_job_as_failed(job.id)
        else:
            mark_job_as_queued(job.id)  # for retry

This is not as powerful as real tools, but it shows the core ideas:

When Not to Use Background Jobs

Sometimes developers move everything to background jobs, which is not always necessary.

Avoid background jobs when:

A good heuristic:

Heuristic: If a task consistently takes more than about 200–500 milliseconds or depends on external services, evaluate if it should be a background job.

Summary

Background jobs allow your backend to:

Key ideas to remember:

In the next chapters about message queues, workers, and specific tools like Celery and Redis, you will see how to implement these background job concepts in real applications.

Views: 16

Comments

Please login to add a comment.

Don't have an account? Register now!

© 2025 Kahibaro
Visitors: 3039 / 24h
Impressum / Imprint
Datenschutz / Data Policy