KAHIBARO
Discord Login Register ☰
☰

32.8. Background Workers

Table of Contents

Why Background Workers Matter in a Production Backend

Most real applications perform tasks that are too slow or too unreliable to run inside the normal HTTP request. Examples:

If you try to do all of this inside your API handler, your responses will be slow, users will retry, and your servers will be fragile.

Background workers solve this by moving work out of the request, into separate processes that can run independently and be scaled separately.

In this chapter you will design how background workers fit into your final project, and what you need to implement to run them reliably in production.

Key idea: Any task that is slow, unreliable, or not critical for the immediate user response should be moved to a background worker.

Role of Background Workers in Your Final Project

For the final production backend, you should explicitly decide:

Common use cases that you can integrate into your project:

Use caseWhy background worker?
Sending registration emailsExternal SMTP latency, possible timeouts
Password reset emailsSame as above, plus rate limiting per user / IP
Generating invoicesCan take time, often CPU heavy (PDF generation)
Image processingCPU heavy, can be batched
Cleaning old sessionsPeriodic maintenance, no user waiting
Syncing with payment APIsExternal API latency, retries needed
Cache warmup & rebuildExpensive DB or search queries, can run in the background

When you design your final project, pick at least 2 or 3 of these to implement with workers, so you exercise:

Architecture: How Workers Integrate with Your System

A typical production setup for background workers in your project:

  1. FastAPI application
    Handles HTTP requests and does only the minimal work needed to respond quickly. When it needs a background task, it publishes a job into a broker (typically Redis).
  2. Message broker (Redis)
    Stores queued jobs. The API writes jobs, workers read jobs. It also stores job states, retries, and schedules (depending on tool).
  3. Worker processes
    Separate processes, often separate containers, that:
    • Subscribes to certain queues
    • Processes jobs
    • Handles retries, backoff, and failure logging
  4. Scheduler (optional but recommended)
    To run periodic jobs, such as hourly or daily maintenance tasks.

A simple conceptual diagram:

In your final deployment, you will likely run:

Choosing a Background Task Tool

You already saw background jobs and queues in earlier chapters. For the final project, choose a tool and stick with it.

Most common Python choices:

ToolBrokerProsCons
CeleryRedis/RabbitMQMature, powerful, widely usedHeavier, more configuration
RQRedisSimple, easy to understandFewer advanced features
DramatiqRedisClean API, good performanceSmaller ecosystem than Celery
FastAPI BackgroundTasksNone (same process)Very simple, no broker neededNot for heavy or long running tasks

For a production‑style final project, prefer a real queue + worker such as:

Using BackgroundTasks from FastAPI is better suited for very small or demo projects because it runs in the same process as the API and does not survive restarts.

Whatever you choose, your architecture will look like:

Designing Your Background Job APIs

You must decide how the rest of your application asks for background work. The main rule:

Your application code should call clean, simple functions like send_welcome_email(user_id)
These functions should not know if a task runs synchronously or in the background.

A typical pattern:

  1. Define a "task" function in a tasks or jobs module.
  2. Expose two interfaces:
    • A Python function to call directly in unit tests or simple environments.
    • A queued version to use in production.

Example design idea (Celery style, simplified code):

python 
# app/tasks/email.py
from .celery_app import celery_app
@celery_app.task(name="send_welcome_email")
def send_welcome_email_task(user_id: int) -> None:
    # Real email sending logic here
    ...
def send_welcome_email(user_id: int, use_background: bool = True) -> None:
    if use_background:
        send_welcome_email_task.delay(user_id=user_id)
    else:
        send_welcome_email_task(user_id=user_id)

Then, in your FastAPI endpoint:

python 
from app.tasks.email import send_welcome_email
@app.post("/register")
def register_user(...):
    user = user_service.register(...)
    send_welcome_email(user.id, use_background=True)
    return {"id": user.id}

During unit tests, you might call send_welcome_email(user.id, use_background=False) so the email logic runs immediately and is easy to assert.

Identifying Tasks to Move to Workers in Your Project

Go through your final project features and list:

  1. Which operations can be slow or unpredictable
  2. Which do not need to finish before the user sees a response
  3. Which can be batched or scheduled

Example for an e‑commerce style project:

FeatureShould be backgrounded?Reason
Send order confirmationYesExternal email service, user does not need to wait
Charge paymentNo (usually)User must know immediately if the payment succeeded
Generate invoice PDFYesCan be slow, user can download later
Update analytics/event logYesNon‑critical, can be processed asynchronously
Update product search indexYesOften slow updates, can lag slightly behind reality

Then, for each "Yes", design:

Queue Design and Priorities

You can have a single queue, or multiple queues with different priorities. For the final project, multiple queues are a good exercise.

Example design:

Queue nameWhat goes herePriority
emailsAll email sending tasksMedium
criticalPayment callbacks, security related tasksHighest
maintenanceCleanup tasks, slow reportsLowest

Practical guidance:

With Celery, you can run workers that listen to specific queues:

bash 
celery -A app.celery_app worker -Q critical,emails -n worker_critical@%h
celery -A app.celery_app worker -Q maintenance -n worker_maintenance@%h

Error Handling, Retries, and Idempotency

In production, some jobs will fail:

Your worker system must handle this automatically.

Retries and Backoff

Most queue systems provide retry configuration. For example:

Example concept with Celery (conceptual code):

python 
@celery_app.task(
    name="send_welcome_email",
    max_retries=5,
    default_retry_delay=30  # seconds
)
def send_welcome_email_task(user_id: int):
    try:
        send_email(...)
    except SomeTemporaryError as exc:
        raise send_welcome_email_task.retry(exc=exc)

Always design background tasks to be idempotent as much as possible.
If a task runs twice, it should not corrupt data.

Idempotency examples:

In your final project, explicitly decide:

Handling Permanent Failures

What happens after all retries fail?

At least:

Ideas for your project:

Monitoring and Observability for Workers

You cannot treat workers as "fire and forget" in production. You need to know:

For your final project, plan:

  1. Health checks
    At least one endpoint or command that tells you if the worker system is healthy. This can be:
    • A specific route in your application that tries a sample Redis operation.
    • A separate process that checks worker status.
  2. Metrics
    At minimum:
    • Number of jobs processed
    • Number of failed jobs
    • Average processing time per job type
    • Queue length

If you integrate Prometheus, you can expose metrics like:

  1. Logging
    Log at least:
    • Start of job with job type and ID
    • Success with duration
    • Failures with stack traces

This is essential when something "mysteriously" stops happening, like "users stopped receiving emails."

Deployment of Workers in Your Production Setup

In your final project, Docker and Compose (or Kubernetes) will orchestrate services. For workers, you typically:

Example docker-compose.yml snippet idea:

yaml 
services:
  api:
    build: .
    command: uvicorn app.main:app --host 0.0.0.0 --port 8000
    depends_on:
      - redis
  worker:
    build: .
    command: celery -A app.celery_app worker -l info
    depends_on:
      - redis
  scheduler:
    build: .
    command: celery -A app.celery_app beat -l info
    depends_on:
      - redis
  redis:
    image: redis:7-alpine

Key points to plan for your project:

Make sure your CI/CD pipeline:

Scheduled and Periodic Jobs

Many background tasks are not triggered by a user action, but by time:

You can implement scheduling in several ways:

ApproachProsCons
External cron + API callsVery simple, language agnosticRequires a server cron or external service
Celery beat or worker schedulerIntegrated with the queue systemMore configuration
Hosted schedulers (e.g. GitHub Actions cron, cloud schedulers)Managed serviceAnother dependency

Example conceptual Celery beat schedule (not full code):

python 
from celery.schedules import crontab
celery_app.conf.beat_schedule = {
    "cleanup-expired-sessions": {
        "task": "cleanup_expired_sessions",
        "schedule": crontab(minute=0, hour="*/1"),  # every hour
    },
    "send-daily-report": {
        "task": "send_daily_report",
        "schedule": crontab(minute=0, hour=0),  # every day at midnight
    },
}

For your final project, pick at least one scheduled job, for example:

This will force you to handle:

Data Access and Transactions in Background Tasks

Background workers still need to access your database, cache, and configuration, just like the API does.

Key rules:

  1. Reuse the same database access layer
    Do not write separate raw SQL for workers. Use the same ORM models and repository classes.
  2. Handle transactions correctly
    A task should:
    • Open a database session
    • Perform its updates
    • Commit or roll back on error
    • Close the session
  3. Be careful with long‑running transactions
    Do not hold a DB transaction open while doing slow external calls. Pattern:
    • Read necessary data from DB, then close the session.
    • Perform external call.
    • Open a new session to update final status.

Example pattern in pseudocode:

python 
def process_invoice_task(invoice_id: int):
    # 1. Load data
    with db_session() as session:
        invoice = repo.get_invoice(session, invoice_id)
        data = build_invoice_data(invoice)
    # 2. Slow external work (no DB transaction open)
    pdf_bytes = generate_pdf(data)
    # 3. Save result
    with db_session() as session:
        repo.save_invoice_pdf(session, invoice_id, pdf_bytes)
        session.commit()

For your final project, review each background task and ensure you do not keep transactions open longer than necessary.

Configuration and Secrets for Workers

Workers need access to the same sensitive configuration as your API:

Rules:

Example variables that both API and worker should read:

Variable nameUsed by
DATABASE_URLAPI, workers
REDIS_URLAPI, workers
SMTP_HOSTWorkers (email tasks)
SMTP_USERWorkers
SMTP_PASSWORDWorkers
APP_ENVAPI, workers (dev/prod)

It is very common to forget that workers need the same configuration as the main app. Plan for this early in your project structure.

Testing Background Workers

For a production‑ready project, only testing HTTP endpoints is not enough. You should also test:

Practical testing strategies:

  1. Unit test the task logic synchronously
    Call the underlying function directly, not through the broker:
python 
   def test_send_welcome_email_sends_email(mocker):
       mock_send = mocker.patch("app.services.email.send_email")
       send_welcome_email(user_id=1, use_background=False)
       mock_send.assert_called_once()
  1. Test that API endpoints add jobs
    For example, patch the .delay method (for Celery) or enqueue method for your chosen library.
  2. Integration testing with a real broker (optional but ideal)
    • Start Redis in your test environment.
    • Start a worker process.
    • Enqueue a job from the test.
    • Wait and then assert that the expected side effect occurred.

Even if full integration tests are complex, unit tests for the task functions are very important.

Putting It All Together for Your Final Project

When you design and implement the final production backend, summarize your background worker plan in a short internal document or README section:

Example checklist:

  1. Tooling
    • [ ] Background processing library chosen (e.g. Celery + Redis)
    • [ ] Common task base module created
  2. Use cases
    • [ ] Registration email implemented as background job
    • [ ] Password reset email implemented as background job
    • [ ] At least one longer running task (e.g. invoice generation) in background
  3. Queues
    • [ ] At least one queue
    • [ ] Optional: separate queues for critical and non‑critical tasks
  4. Retries and idempotency
    • [ ] Retry policy configured per task
    • [ ] All tasks safe to rerun or protected by idempotency logic
  5. Scheduling
    • [ ] At least one scheduled job (e.g. cleanup)
  6. Deployment
    • [ ] Worker container defined in Docker Compose or Kubernetes
    • [ ] Same image and configuration as API
    • [ ] Easy to scale worker count
  7. Monitoring
    • [ ] Logs for job start, success, failure
    • [ ] Basic metrics or at least queue length visibility
    • [ ] Health checks or manual commands to verify workers are alive

By following this plan, your final project will not just have background tasks as a side feature. It will have a real, production‑style background processing system, with proper architecture, configuration, deployment, and monitoring.

Views: 8

Comments

Please login to add a comment.

Don't have an account? Register now!

© 2025 Kahibaro
Visitors: 3516 / 24h
Impressum / Imprint
Datenschutz / Data Policy