KAHIBARO
Discord Login Register ☰
☰

17.9. Email Jobs

Table of Contents

Why Email Belongs in Background Jobs

Backend applications send many types of emails: verifications, password resets, receipts, notifications, and more. If you send emails directly inside a request handler, you create several problems:

This is why email is a perfect example of a background job. The API accepts the request quickly, schedules an email job, and returns a response. A separate worker sends the email in the background.

Rule: Never block a user-facing request on a slow external service like an email provider. Offload it to a background job whenever possible.

In this chapter, you will see practical patterns for implementing email jobs using queues and workers, and how to make them reliable and safe.

Synchronous vs Background Email Sending

Synchronous Email Example

Imagine a typical registration handler written in a synchronous style:

python 
def register_user(form_data):
    user = create_user_in_db(form_data)
    # Synchronous email sending
    send_verification_email(user.email)
    return {"message": "Registration complete, check your email"}

The request is blocked until send_verification_email finishes:

If this takes 2 seconds, your registration endpoint always takes at least 2 seconds. If the email provider times out, the whole request might fail and the user thinks they did not register at all.

Background Email Example

With a background job, the handler becomes:

python 
def register_user(form_data):
    user = create_user_in_db(form_data)
    # Schedule email job
    enqueue_email_job(
        "send_verification_email",
        to=user.email,
        token=user.verification_token
    )
    return {"message": "Registration complete, check your email soon"}

Then a worker process consumes the job:

python 
def worker_loop():
    while True:
        job = queue.get()
        if job.name == "send_verification_email":
            send_verification_email(job.args["to"], job.args["token"])

The user response is fast, and email delivery is decoupled from the HTTP request.

Types of Email Jobs

Different emails have different timing and reliability requirements. It helps to think of them in categories.

Transactional Emails

Transactional emails are directly triggered by a user action.

Common examples:

Characteristics:

These are often placed on a high-priority queue with retries.

Batch or Digest Emails

Batch or digest emails are periodic summaries or grouped notifications.

Examples:

Characteristics:

These are well suited to scheduled jobs that generate a list of recipients and enqueue individual email jobs.

System and Admin Notifications

Admin or system-related emails:

Characteristics:

These might use a separate queue or channel so they are not delayed by user traffic.

Basic Email Job Flow

A common flow for an email job is:

  1. Trigger
    An event occurs, for example, user registers, user requests password reset, order is placed.
  2. Job creation
    The backend creates a job description, often a small JSON-like payload:
json 
   {
     "job_type": "send_verification_email",
     "to": "alice@example.com",
     "user_id": 123,
     "verification_token": "abcd1234",
     "created_at": "2026-08-28T12:34:56Z"
   }
  1. Queueing
    The job is pushed to a message queue (for example Redis list, RabbitMQ, SQS, etc.).
  2. Worker processing
    A worker process reads jobs from the queue and:
    • Builds the email content (subject, body, HTML).
    • Sends the email through SMTP or an email provider API.
    • Logs success or failure.
    • Optionally updates the database (for example, stores "email_sent_at").
  3. Retries
    If sending fails, the worker retries a few times with backoff.

Email Job Payload Design

Your job payload should include enough data to send the email, but not too much.

Two Options

  1. Store most data in the job

Example payload:

json 
   {
     "job_type": "send_order_confirmation",
     "to": "alice@example.com",
     "order_id": 101,
     "order_total": 49.90,
     "items": [
       {"name": "Book A", "qty": 1, "price": 29.90},
       {"name": "Book B", "qty": 1, "price": 20.00}
     ]
   }

Pros:

Cons:

  1. Store references, load data in worker

Example payload:

json 
   {
     "job_type": "send_order_confirmation",
     "to": "alice@example.com",
     "order_id": 101
   }

The worker fetches order details from the database.

Pros:

Cons:

Practical Guideline

json 
  {
    "job_type": "send_password_reset",
    "to": "alice@example.com",
    "user_id": 123,
    "reset_token": "xyz"
  }

Using Redis Queues for Email Jobs

Redis is frequently used as a lightweight job queue. Many Python libraries build on Redis to provide higher-level job abstractions.

Although the exact implementation details belong in other chapters, here is a conceptual example using a Redis list as a simple queue.

Enqueuing an Email Job

python 
import json
import redis
from datetime import datetime, timezone
redis_client = redis.Redis(host="localhost", port=6379, db=0)
def enqueue_email_job(job_type, to, payload):
    job = {
        "job_type": job_type,
        "to": to,
        "payload": payload,
        "created_at": datetime.now(timezone.utc).isoformat()
    }
    redis_client.rpush("email_jobs", json.dumps(job))
# Example usage
enqueue_email_job(
    job_type="send_verification_email",
    to="alice@example.com",
    payload={"token": "abcd1234"}
)

This code pushes a JSON-encoded job onto the email_jobs list.

Worker Consuming Email Jobs

python 
import json
import time
import redis
redis_client = redis.Redis(host="localhost", port=6379, db=0)
def send_verification_email(to, token):
    # Pseudocode: integrate your real email sending here
    print(f"Sending verification email to {to} with token {token}")
def process_job(job):
    job_type = job["job_type"]
    to = job["to"]
    payload = job["payload"]
    if job_type == "send_verification_email":
        send_verification_email(to, payload["token"])
    else:
        # Unknown job type, log or discard
        print(f"Unknown job type: {job_type}")
def worker_loop():
    while True:
        # BLPOP blocks until there is a job
        _, job_data = redis_client.blpop("email_jobs")
        job = json.loads(job_data)
        try:
            process_job(job)
        except Exception as exc:
            print(f"Failed to process job: {exc}")
            # Optionally move to a dead-letter queue or retry
            redis_client.rpush("email_jobs_failed", job_data)
            time.sleep(1)
if __name__ == "__main__":
    worker_loop()

In a real system you would replace the print with SMTP or provider API calls and logging.

Email Templates in Jobs

Building email content inside the worker allows you to reuse templates and keep your HTML out of business logic.

Simple Template Example

python 
from string import Template
VERIFICATION_TEMPLATE = Template("""
Hello,
Please verify your email address by clicking this link:
$verification_link
If you did not create an account, you can ignore this message.
Thanks,
The Example App Team
""".strip())
def build_verification_email(to, token):
    verification_link = f"https://example.com/verify?token={token}"
    subject = "Verify your email"
    body = VERIFICATION_TEMPLATE.substitute(verification_link=verification_link)
    return subject, body

Worker usage:

python 
def send_verification_email(to, token):
    subject, body = build_verification_email(to, token)
    send_email_smtp(to=to, subject=subject, body=body)

You can expand this to handle HTML + text versions, localization, or different brands.

Retrying Failed Email Jobs

External email providers can fail due to network issues or temporary problems. A background job system can retry automatically.

Retry Strategy

You can define a policy like:

Exponential backoff example formula:

$$
t_n = t_0 \times 2^{(n-1)}
$$

Where:

Rule: Always limit the maximum number of retries and implement backoff. Infinite rapid retries can overload your system and the email provider.

Simple Retry Example with Metadata

Add retry metadata to the job:

json 
{
  "job_type": "send_verification_email",
  "to": "alice@example.com",
  "payload": {"token": "abcd1234"},
  "retries": 0
}

Worker logic:

python 
import time
import json
import redis
MAX_RETRIES = 5
BASE_DELAY = 5  # seconds
redis_client = redis.Redis(host="localhost", port=6379, db=0)
def schedule_retry(job):
    job["retries"] += 1
    delay = BASE_DELAY * (2 ** (job["retries"] - 1))
    print(f"Retrying in {delay} seconds")
    time.sleep(delay)  # Simple approach; many systems use separate delay queues
    redis_client.rpush("email_jobs", json.dumps(job))
def worker_loop():
    while True:
        _, job_data = redis_client.blpop("email_jobs")
        job = json.loads(job_data)
        try:
            process_job(job)
        except Exception as exc:
            print(f"Error processing job: {exc}")
            if job.get("retries", 0) < MAX_RETRIES:
                schedule_retry(job)
            else:
                print("Max retries reached. Moving to failed queue.")
                redis_client.rpush("email_jobs_failed", json.dumps(job))

In production you would not use time.sleep inside the worker loop for delays at scale, but the concept is the same: next attempt is scheduled for later.

Idempotency in Email Jobs

Sometimes the same job might get processed twice, for example:

If your job sends an email twice, the user might receive two identical messages.

For some emails this is acceptable, but for others (like password reset codes) it can be confusing.

Idempotency Concept

A job is idempotent if running it multiple times has the same effect as running it once.

For email jobs, full idempotency is hard because sending the same email twice still sends twice. But you can limit duplicates per event.

Practical Patterns

  1. Store a unique email event ID
    • Generate an email_event_id when enqueuing the job.
    • Before sending, the worker checks a table like email_events to see if that event ID was already processed.
    • If yes, skip.
    • If no, send and record the event ID.
  2. Store “last sent” timestamps

For frequent notifications:

  1. Reasonable acceptance of duplicates

For some emails, it is simpler to accept that occasional duplicates are not critical, and rely on clear content like:

If you already used a more recent reset link, you can ignore this email.

Using Background Jobs from an API (Example)

Here is a simple illustration of how an HTTP endpoint might schedule an email job. This is conceptual and focuses on the job aspect, not on full framework details.

Password Reset Request Endpoint

python 
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, EmailStr
app = FastAPI()
class PasswordResetRequest(BaseModel):
    email: EmailStr
def create_reset_token_for_email(email: str) -> str:
    # Pseudocode: check user exists, create token, save in DB
    return "generated-reset-token"
@app.post("/password-reset")
def request_password_reset(data: PasswordResetRequest):
    token = create_reset_token_for_email(data.email)
    enqueue_email_job(
        job_type="send_password_reset",
        to=data.email,
        payload={"token": token}
    )
    # Response does not wait for email sending
    return {"message": "If an account exists for this email, a reset link has been sent."}

Worker:

python 
def send_password_reset_email(to, token):
    reset_link = f"https://example.com/reset-password?token={token}"
    subject = "Reset your password"
    body = f"Click the link to reset your password: {reset_link}"
    send_email_smtp(to=to, subject=subject, body=body)
def process_job(job):
    job_type = job["job_type"]
    to = job["to"]
    payload = job["payload"]
    if job_type == "send_password_reset":
        send_password_reset_email(to, payload["token"])
    # handle other job types here

This separates:

Scheduling Email Jobs

Some emails are not triggered by an immediate HTTP request, but by time-based schedules.

Typical scheduled email jobs:

Simple Conceptual Flow

  1. A scheduler (for example, cron, Celery beat, a custom scheduler) runs a function every day at 09:00.
  2. This function:
    • Queries the database for users who should receive that email.
    • For each user, enqueues an email job.
  3. Existing workers process the queued email jobs.

Example scheduler function:

python 
def schedule_daily_digests():
    users = get_users_to_receive_digest()
    for user in users:
        enqueue_email_job(
            job_type="send_daily_digest",
            to=user.email,
            payload={"user_id": user.id}
        )

Worker implementation:

python 
def send_daily_digest_email(to, user_id):
    items = get_daily_items_for_user(user_id)
    subject = "Your daily summary"
    body = build_daily_digest_body(items)
    send_email_smtp(to=to, subject=subject, body=body)

Security and Privacy Considerations

Emails contain sensitive information and are often used for authentication flows.

Important Rules

Rule: Never include secrets like raw passwords or full credit card numbers in an email.
Rule: For actions like password reset or email verification, use short-lived, single-use tokens instead of embedding user IDs or credentials directly.
Rule: Avoid logging full email contents or tokens in plain text logs.

Common Mistakes to Avoid

Observability for Email Jobs

To operate email jobs in production, you need visibility.

Key metrics to track:

MetricDescription
Enqueued jobs countHow many email jobs are created
Processed jobs countHow many were processed successfully
Failed jobs countHow many ended up in the dead-letter queue
Average send timeTime from enqueue to successful send
Queue lengthHow many jobs are waiting
Provider error ratePercentage of email API errors

Simple logging idea in worker:

python 
def process_job(job):
    start = time.time()
    try:
        # send email
        ...
        duration = time.time() - start
        print(f"Email job {job['job_type']} to {job['to']} succeeded in {duration:.2f}s")
    except Exception as exc:
        print(f"Email job failed: {exc}")
        # retry or move to dead-letter queue

In more advanced systems you would send these metrics to dedicated monitoring tools.

Summary

In this chapter you saw how email fits naturally into background processing:

You now have a clear picture of how backend applications typically handle email as background work, and how it connects with message queues, workers, and reliability patterns.

Views: 16

Comments

Please login to add a comment.

Don't have an account? Register now!

© 2025 Kahibaro
Visitors: 2995 / 24h
Impressum / Imprint
Datenschutz / Data Policy