KAHIBARO
Discord Login Register ☰
☰

18.7. Background Email Delivery

Table of Contents

Why Send Emails in the Background?

Whenever your backend sends an email, it usually talks to an SMTP server over the network. That can be slow and unreliable. If you send emails directly inside your API handler:

Typical examples:

In all these cases, the user should not have to wait for the email. The API should respond quickly, and the email should be sent in the background.

Important rule: Do not block user requests while sending emails. Use background processing so responses stay fast and reliable.

Background email delivery solves this by:

Synchronous vs Background Email Sending

Synchronous Email Sending

Synchronous means “do it now, in this request.”

In a typical API endpoint:

python 
def register_user(request):
    user = create_user(request)
    send_verification_email(user.email)  # slow operation
    return {"message": "User created"}

Problems with this approach:

For small side projects or local testing, synchronous sending is fine. But it does not scale.

Background Email Sending

Background sending separates “accepting the request” from “doing the work.” The endpoint only describes the work to do and returns, and a background worker does the actual sending.

Conceptually:

python 
def register_user(request):
    user = create_user(request)
    enqueue_email_job(
        to=user.email,
        subject="Verify your account",
        template="verify.html",
        context={"token": user.verification_token},
    )
    return {"message": "User created"}

A separate worker:

python 
def worker_loop():
    while True:
        job = get_next_email_job()
        send_email(job.to, job.subject, job.template, job.context)

Benefits:

Key idea: Treat every email as a job that can be queued, retried, and processed by a worker, instead of something that must complete before returning an API response.

Background Email Patterns

There are several common patterns to implement background email delivery. The core idea is always “queue work, then process it.”

Pattern 1: Fire-and-Forget in the Application Server

Some frameworks let you schedule “background tasks” inside the same process that handles HTTP requests.

Example idea:

python 
@app.post("/register")
def register(user_data, background_tasks: BackgroundTasks):
    user = create_user(user_data)
    background_tasks.add_task(
        send_verification_email,
        user.email,
        user.verification_token
    )
    return {"message": "User created"}

Characteristics:

Limitations:

Use this when:

Pattern 2: Message Queue + Worker

A more robust pattern uses:

High-level flow:

  1. API receives a request that should trigger an email.
  2. API creates a job description (JSON object) and pushes it into a queue.
  3. The API returns a response immediately.
  4. A worker process constantly reads jobs from the queue.
  5. For each job, the worker calls send_email.
  6. If sending fails, the worker can retry or move the job to a “dead letter” queue.

Typical job structure:

json 
{
  "type": "send_email",
  "to": "user@example.com",
  "subject": "Reset your password",
  "template": "password_reset.html",
  "context": {
    "reset_link": "https://example.com/reset?token=abc123"
  },
  "attempt": 1
}

Benefits:

This is a very common pattern for production email delivery.

Pattern 3: Task Queue Framework (Celery, RQ, etc.)

Task queue frameworks sit on top of message queues and give you:

Conceptually, you define a background task:

python 
@task
def send_verification_email_task(to_email, token):
    send_verification_email(to_email, token)

Then call it from your API:

python 
def register_user(request):
    user = create_user(request)
    send_verification_email_task.delay(user.email, user.verification_token)
    return {"message": "User created"}

The framework handles:

This is a common choice for real projects because it gives you a lot of functionality with minimal boilerplate.

What to Put in an Email Job

A background “email job” should contain enough information to build and send the email, but not too much. You have two main options.

Option 1: Store Full Email Data

Store everything needed:

Example job:

json 
{
  "to": "user@example.com",
  "subject": "Welcome to Example",
  "template": "welcome.html",
  "context": {
    "first_name": "Alice",
    "dashboard_url": "https://example.com/dashboard"
  }
}

Worker:

python 
def process_email_job(job):
    html = render_template(job["template"], job["context"])
    send_raw_email(job["to"], job["subject"], html)

Pros:

Cons:

Use this when:

Option 2: Store References, Fetch Data Later

Store only IDs or references, and let the worker fetch data from your database.

Example job:

json 
{
  "user_id": 123,
  "type": "welcome_email"
}

Worker:

python 
def process_email_job(job):
    user = get_user_by_id(job["user_id"])
    subject, html = build_welcome_email(user)
    send_raw_email(user.email, subject, html)

Pros:

Cons:

Use this when:

Rule of thumb: Store only what you need to build the email reliably. Avoid putting secrets or large payloads into the queue unless absolutely necessary.

Retry Strategies for Email Jobs

Email sending is often unreliable. SMTP servers can:

Background processing makes it easier to retry logically instead of immediately.

Basic Retry With Counter

Each job keeps track of how many attempts have been made.

Example job fields:

json 
{
  "to": "user@example.com",
  "subject": "Verify your account",
  "template": "verify.html",
  "context": {"token": "abc123"},
  "attempt": 1
}

Worker logic (conceptual):

python 
MAX_ATTEMPTS = 5
def process_email_job(job):
    try:
        send_email(job)
    except TemporaryEmailError:
        if job["attempt"] < MAX_ATTEMPTS:
            job["attempt"] += 1
            requeue_with_delay(job, delay=calculate_delay(job["attempt"]))
        else:
            move_to_dead_letter(job)
    except PermanentEmailError:
        move_to_dead_letter(job)

Exponential Backoff

You usually do not want to retry every few seconds forever. A common pattern is “exponential backoff,” where you wait longer after each failure.

Example formula:

$$
\text{delay\_seconds} = \min(2^{n} \times 10, 3600)
$$

Where:

So the delays are: 20s, 40s, 80s, 160s, 320s, etc, capped at 1 hour.

Exponential backoff rule: Increase the delay after each failed attempt so you do not overwhelm the SMTP server or your own system with rapid retries.

Dead Letter Queue

After too many failures, an email job should not be retried automatically.

Instead:

This lets you:

Temporary vs Permanent Errors

Classify errors so that you can decide if you should retry.

Examples:

Error typeExampleRetry?
Network timeoutSMTP server did not respondYes
4xx temporary SMTP code“Mailbox temporarily unavailable”Yes
DNS lookup failureCannot resolve SMTP hostYes
5xx permanent “user unknown”Address does not existNo
Invalid “to” email format“abc@@example.com”No
Misconfigured credentialsWrong SMTP passwordPossibly

Treat temporary errors with retries and exponential backoff, and treat permanent errors as failures that go to the dead letter queue.

Idempotency and Duplicate Emails

Sometimes your background system can process the same job twice. For example:

If your sending logic is naive, the user may receive 2 identical emails.

To avoid this, you can add idempotency to your email sending.

Using a Unique Email Job ID

Give each email job a unique identifier, and store a record that it was sent.

Example table email_log:

idjob_idtypeto_emailsent_at
1"reset-123-001""passwordReset""user@example.com"2026-08-20 10:01:02

Process logic:

python 
def process_email_job(job):
    if email_log_exists(job["job_id"]):
        # Already sent, do nothing
        return
    send_email(job)
    save_email_log(job["job_id"], job["type"], job["to"])

If the queue delivers the same job again, the worker checks the log and skips sending.

Idempotent Design at Higher Level

Sometimes you may design at a higher level:

This does not fully prevent duplicates, but it reduces the harm if they happen.

Idempotency rule: Email jobs should be designed so that processing the same job more than once does not result in unexpected or harmful behavior.

Prioritizing and Categorizing Emails

Not all emails are equal. Some must be sent immediately, others can wait.

Typical categories:

In a background system, you can:

Example:

Queue namePurposeWorkers
email_highPassword resets, verification5
email_mediumNotifications3
email_lowNewsletters1

Your API code decides which queue to use based on email type.

This prevents a huge newsletter sending from delaying critical password reset messages.

Observability: Logging and Monitoring Email Jobs

Background email delivery should be observable so you can:

Logging

Log important events:

Example log entries:

text 
INFO  Created email job 123 type=verification to=user@example.com
INFO  Sent email job 123 to=user@example.com in 0.82s
ERROR Failed email job 124 to=user@example.com attempt=3 error="SMTP timeout"

Useful logging fields:

FieldDescription
job_idUnique ID for the job
typeEmail type, like verification
to_emailRecipient address (maybe masked)
attemptAttempt number
duration_msTime taken to send
errorError message or exception type

Metrics

Track metrics so you can see trends:

These help you answer questions like:

Alerts

Set alerts for conditions such as:

Then you can react quickly before users start complaining that they are not receiving emails.

Common Pitfalls in Background Email Delivery

When building background email systems, watch out for some typical mistakes.

Pitfall 1: Sending Email Before Commit

If email sending is triggered before your database transaction commits, you can send emails that refer to data that does not exist.

Example:

  1. Start transaction.
  2. Insert user into database.
  3. Enqueue email job.
  4. Transaction fails and is rolled back.
  5. Worker sends email with a verification link that no longer works.

Better approach:

Pitfall 2: Missing Error Handling in Workers

If your worker loop does not handle exceptions, a single failing email can crash the worker, and no more emails are sent.

Always wrap job processing in a try/except that:

Pitfall 3: Mixing Business Logic Inside the Worker

If your worker contains too much business logic, it becomes another “backend” to maintain.

Keep workers small:

This makes the system easier to test and reason about.

Pitfall 4: Not Handling SMTP Limits

Many email providers have limits, for example:

Your worker should respect these limits, for example:

Background processing makes this easier because you can control how many workers you run and how fast they pull jobs.

Putting It All Together: Typical Flow

Here is a complete picture of how background email delivery often works in a backend application.

  1. User action
    A user triggers an event that requires an email, for example user registration.
  2. Application logic
    Your API:
    • Validates input.
    • Writes data to the database.
    • Commits the transaction.
  3. Enqueue email job
    After commit, your code:
    • Creates an email job object.
    • Chooses a queue (for example email_high).
    • Pushes the job into the queue.
  4. Immediate response
    The API returns a success response to the user without waiting for the email.
  5. Worker processing
    In the background:
    • A worker reads a job from the queue.
    • Optionally checks idempotency (has this job already been sent?).
    • Builds the email content.
    • Calls the email sending library to send it via SMTP or an email service API.
    • Logs the result.
  6. Retries and errors
    If sending fails:
    • The worker classifies the error as temporary or permanent.
    • Temporary: increases the attempt count and requeues with exponential backoff.
    • Permanent or too many attempts: moves to dead letter queue and logs error.
  7. Monitoring
    Metrics and logs let you observe:
    • How many emails are sent.
    • How many fail.
    • Whether the queue is growing.

This design keeps your user-facing API fast and improves reliability, while still delivering emails in a robust and controlled way.

Summary rule: Handle email as asynchronous jobs with clear job data, retries with exponential backoff, idempotency, and proper logging, instead of sending directly in user requests.

Views: 19

Comments

Please login to add a comment.

Don't have an account? Register now!

© 2025 Kahibaro
Visitors: 3039 / 24h
Impressum / Imprint
Datenschutz / Data Policy