KAHIBARO
Discord Login Register ☰
☰

17.10. Long-Running Tasks

Table of Contents

Why Long-Running Tasks Are a Problem in Web Backends

In a web backend you usually handle HTTP requests. A user clicks a button, your server receives a request, does some work, and returns a response.

This flow breaks down when the work takes a long time, for example:

If you try to do this work directly inside the HTTP request handler, several problems appear:

Long‑running tasks need a different approach.

Types of Long-Running Tasks

Long‑running tasks come in several patterns. It is useful to recognize them because each type might be handled a bit differently.

CPU-heavy tasks

These use a lot of CPU:

They may block threads and reduce the number of requests your app can handle. You often want to move these away from your main web process to dedicated workers.

I/O-heavy tasks

These spend most of the time waiting on input and output:

Sometimes asynchronous I/O can help, but if a single operation takes a long time (for example a 5 minute external API call), you still want to move it to a background worker instead of blocking the request.

Batch processing tasks

These work on a lot of items as a group:

They are scheduled or triggered by some condition, and can take minutes or hours. They are usually run as background jobs or scheduled tasks, not during HTTP requests.

User-triggered long actions

The user starts an action that cannot finish quickly:

Here you want:

  1. The HTTP request to respond quickly.
  2. The heavy work to continue in the background.
  3. A way for the user to check progress and download the result later.

Patterns for Handling Long-Running Tasks

You rarely want to keep a user waiting for a long‑running task to finish inside a single request. Instead, use patterns that separate the trigger from the execution.

Fire-and-forget (not recommended in most cases)

The simplest idea is to start a background thread or async task inside your web process and immediately return a response.

Example in Python using a thread:

python 
from threading import Thread
def send_report_email(user_id):
    # heavy work here
    ...
@app.post("/send-report")
def send_report(user_id: int):
    Thread(target=send_report_email, args=(user_id,)).start()
    return {"status": "scheduled"}

Problems:

Useful only for very small projects or prototypes.

Important rule: For reliable long‑running tasks in production, do not rely on ad‑hoc threads or in-process background tasks. Use a proper job queue and workers.

Job queue and workers

A more robust pattern is:

  1. The HTTP request handler creates a job and pushes it to a queue.
  2. A separate worker process listens to the queue, pulls jobs, and executes them.
  3. The web process returns quickly, usually with a job id.
  4. The client can later check job status or see the result.

Typical components:

Basic flow with a job queue:

text 
Client -> API server -> enqueue job in Redis -> return job_id
                                   |
                               Worker process
                              reads job from
                                   Redis
                                   |
                                 executes
                                   |
                        updates job status / result

Polling and callbacks

After you enqueue a long‑running job, you need a way to notify the user or client.

Two main options:

  1. Client polling
    • Client receives a job_id.
    • Client calls an endpoint like GET /jobs/{job_id} every few seconds.
    • Backend returns status: pending, running, failed, completed, and possibly progress.
  2. Server callback / webhook
    • Client gives you a callback URL.
    • When the job finishes, your backend calls that URL with the result.
    • Often used for server‑to‑server integrations.

Table comparison:

PatternProsCons
PollingEasy to implement, no extra infraExtra requests, status may be slightly stale
CallbackInstant notifications, fewer requestsRequires public endpoint and security checks

Chunking work

Sometimes a single job is so big that you want to split it into smaller jobs.

For example:

Advantages:

You usually track an overall “parent” job that knows how many sub-jobs must finish.

Designing APIs Around Long-Running Tasks

As a backend developer you must design HTTP APIs that safely trigger and manage long‑running tasks.

Pattern: Accept-and-track

Step 1, the client sends a request to start the work.

Example:

http 
POST /reports
Content-Type: application/json
{
  "user_id": 123,
  "month": "2026-05"
}

Step 2, the backend:

  1. Validates the input.
  2. Creates a job record in the database, for example:

| id | type | status | created_at | progress | result_url |
|-----|-----------|----------|---------------------|----------|------------|
| 42 | "report" | "queued" | 2026-05-01 10:00:00 | 0 | null |

  1. Publishes a job to the queue with job_id = 42.
  2. Returns a 202 Accepted response, not 200 OK, to signal that the request is accepted but not completed.

Response:

http 
HTTP/1.1 202 Accepted
Content-Type: application/json
{
  "job_id": 42,
  "status": "queued",
  "status_url": "/jobs/42"
}

Step 3, the client polls:

http 
GET /jobs/42

Possible responses:

json 
{
  "job_id": 42,
  "status": "running",
  "progress": 37
}

or

json 
{
  "job_id": 42,
  "status": "completed",
  "result_url": "/downloads/report-42.pdf"
}

or

json 
{
  "job_id": 42,
  "status": "failed",
  "error": "Could not access data source"
}

Why 202 Accepted is useful

When you cannot complete an operation in a single request, 202 Accepted is a clear signal:

This is more correct than returning 200 OK with an empty result for operations that are still running.

Important rule: For long‑running operations triggered over HTTP, prefer 202 Accepted plus a status endpoint instead of waiting for the full task to complete.

Simple example design

Imagine a “Generate purchase history CSV” feature:

Your backend:

  1. Creates an Export record with fields like id, user_id, status, created_at, file_path.
  2. Enqueues a job that contains export_id.
  3. Worker generates CSV, stores file, updates Export record with status = 'completed' and file_path.
  4. Download endpoint checks status and returns the file only when ready.

This pattern will appear again and again in real‑world backends.

Timeouts, Limits, and Reliability

Long‑running tasks are very sensitive to limits and failures. Design with them in mind from the start.

Know your time limits

Different parts of your stack may have limits:

For example, a worker task might be killed if it runs longer than 10 minutes. You must tune these values realistically or split tasks into smaller parts.

If a task often approaches your time limit, consider:

Handling failures and retries

Long‑running tasks touch many external systems, so failures are common:

You rarely want to fail permanently on the first error. Instead you use retries, often with delay and backoff.

Common strategy:

Basic exponential backoff formula:

$$
\text{delay}_n = \text{base} \times 2^{n-1}
$$

Example: base = 1 second.

Important rule: Long‑running tasks should be idempotent or safely retryable. A retry must not create duplicate data or corrupt state.

Idempotency usually means:

Progress tracking

For very long tasks, users want to see progress, not only “running”.

Simple progress tracking strategies:

In code, you might store progress in a database table or cache:

job_idstatusprocessedtotallast_update
42running25310002026-05-01 10:05:00

Workers update this record periodically. The status endpoint reads it and returns structured data to the client.

Example: End-to-End Flow for a Long-Running Task

Let us put everything together in an example scenario.

Goal: User uploads a big CSV file with 100,000 rows. Backend should import and validate the data. This may take 5 minutes.

Step 1: User uploads CSV

| id | user_id | status | file_path | progress | total_rows |
|----|---------|----------|--------------------|----------|-----------|
| 7 | 123 | queued | /tmp/import7.csv | 0 | null |

json 
    {
      "job_id": 7,
      "status": "queued",
      "status_url": "/imports/7"
    }

Step 2: Worker runs the job

Pseudo code:

python 
def import_csv(job_id: int):
    job = get_job(job_id)
    job.status = "running"
    save_job(job)
    with open(job.file_path) as f:
        rows = list(read_rows(f))
    job.total_rows = len(rows)
    save_job(job)
    for index, row in enumerate(rows, start=1):
        try:
            process_row(row)
        except ValidationError as e:
            log_error(job_id, row, str(e))
        # update progress every 100 rows to reduce writes
        if index % 100 == 0 or index == job.total_rows:
            job.progress = index
            save_job(job)
    job.status = "completed"
    save_job(job)

Step 3: Client polls for status

Every few seconds:

http 
GET /imports/7

Possible responses:

While running:

json 
{
  "job_id": 7,
  "status": "running",
  "processed": 4200,
  "total": 100000
}

When finished:

json 
{
  "job_id": 7,
  "status": "completed",
  "processed": 100000,
  "total": 100000,
  "errors_url": "/imports/7/errors"
}

If something fails:

json 
{
  "job_id": 7,
  "status": "failed",
  "error": "Database is temporarily unavailable. Please retry later."
}

The web server remains responsive during the whole 5 minute import, because the heavy work runs in worker processes.

Practical Tips and Common Pitfalls

Tips

Pitfalls

What You Should Take Away

You will use these patterns across many backend projects, from simple report generation to complex data pipelines.

Views: 8

Comments

Please login to add a comment.

Don't have an account? Register now!

© 2025 Kahibaro
Visitors: 3512 / 24h
Impressum / Imprint
Datenschutz / Data Policy