17.3. Message Queues

Table of Contents

Why Message Queues Matter

In many backend systems you do not want to do everything inside a single HTTP request. Some tasks are slow, unreliable, or involve other services that may fail or be busy. Message queues give you a way to move work out of the request, process it in the background, and build systems that are more reliable and easier to scale.

At a simple level, a message queue is:

A place where producers put messages (pieces of work).
A place where workers take messages, process them, and remove them.

This sounds simple, but it is one of the most important building blocks in modern backend systems.

Core Concepts

Producers and Consumers

In a message queue system you will usually see these roles:

Role	Also called	Responsibility
Producer	Publisher, Sender	Creates messages and sends them to a queue
Consumer	Worker, Subscriber	Receives messages and processes them
Broker	Queue server	Stores messages and delivers them to consumers

The broker is the service that implements queues. Examples are:

RabbitMQ
Redis (used as a simple broker)
AWS SQS
Kafka (more of a streaming platform, but often used similarly)

Simple Example

Imagine your API has an endpoint:

http

POST /send-welcome-email

Without a queue, the handler might:

Validate the request
Render the email template
Connect to an SMTP server
Send the email
Return 200 OK

If step 3 or 4 is slow, the user waits. With a queue:

Validate the request
Create a message: { "type": "send_welcome_email", "user_id": 123 }
Publish the message to a queue "emails"
Return 202 Accepted quickly
A worker process reads from "emails" and sends the actual email

The API is now faster and more reliable.

What Is a Message?

A message is a unit of data that describes a task or event. It is usually a small JSON object or a binary payload.

Typical fields:

Field	Purpose
`id`	Unique identifier for tracking
`type`	Type of job or event
`payload`	Data needed to do the work
`created_at`	When the message was created
`retry`	How many times it has been retried so far

Example message for sending an email:

json

{
  "id": "79b3a0b8-672c-4e7f-9c11-3b5dc43de4ee",
  "type": "send_email",
  "payload": {
    "to": "user@example.com",
    "subject": "Welcome!",
    "template": "welcome_email",
    "data": {
      "name": "Alice"
    }
  },
  "created_at": "2026-08-27T12:34:56Z",
  "retry": 0
}

The broker does not care what the message means. It just stores and delivers it. Producers and consumers agree on the message format.

How Message Queues Work

Basic Operations

Even though different systems have different APIs, the core operations are usually:

Operation	What it does
`publish`	Send a message to a queue
`consume`	Wait for a message and receive it
`ack`	Acknowledge that a message was processed successfully
`reject/nack`	Tell the broker that processing failed, possibly requeue the message
`peek`	View a message without removing it (not always supported)

In many libraries you will not call ack directly. The library will do it for you when your worker function returns without error.

Conceptual Flow

Producer publishes message M to queue Q
Broker stores M
Consumer subscribes to Q
Broker delivers M to the consumer
Consumer processes M
Consumer acknowledges M
Broker removes M from Q

If the consumer crashes before step 6, the broker can give the same message to another consumer after a timeout.

Important rule: A message should be processed at least once. You must design your consumers to handle the same message more than once without causing incorrect results. This is called idempotency.

Queues vs Topics vs Streams

You will see different patterns:

Pattern	Description	Typical tech
Queue	Each message is processed by one consumer	RabbitMQ, Redis list
Topic	Each message is broadcast to many subscribers	Pub/Sub, RabbitMQ
Stream	Messages are appended to a log, consumers read at their own pace	Kafka, Redis Streams

For background jobs in a web backend, a simple queue is usually enough.

Common Use Cases

Backend developers use message queues for many scenarios.

Offloading Slow Tasks

You move slow operations away from the request.

Examples:

Sending emails or SMS
Generating PDFs or reports
Resizing images
Uploading large files to another service
Calling third party APIs (payment, analytics)

Example HTTP flow with queue:

Client calls POST /generate-report
API validates parameters
API enqueues job { "type": "generate_report", "user_id": 123 }
API returns 202 Accepted and maybe a report_id
A worker generates the report and updates report status in the database

The client can later call GET /reports/{report_id} to check status.

Decoupling Services

If you have multiple services, a queue can decouple them.

Scenario:

Service A handles user registration
Service B sends welcome emails
Service C logs analytics

Instead of A calling B and C directly, A puts a message "user_registered" on a queue or topic. B and C subscribe and react independently.

Benefits:

If B is down, A still works. Messages stay in the queue until B is back.
You can add new consumers (e.g. Service D that gives bonus points) without changing A.

Rate Limiting and Throttling Work

If an external API only allows 100 requests per minute, you can:

Put all jobs that call that API into a queue.
Have a worker that reads from the queue and limits processing speed (e.g. one job every 600 ms).

The queue absorbs bursts. Your API remains responsive, while the work is processed at a safe speed.

Message Queue Guarantees

Different systems provide different delivery guarantees. You must understand these to design correct systems.

Guarantee	Meaning
At most once	Each message is delivered 0 or 1 times. Messages can be lost, never duplicated.
At least once	Each message is delivered 1 or more times. No loss, but possible duplicates.
Exactly once	Each message is delivered exactly 1 time, from the consumer's view. Very hard and rare.

Most common brokers give at least once guarantee by default.

Important rule: With "at least once" delivery, always design workers to be idempotent. Processing the same message multiple times must not cause wrong data or double billing.

Idempotent Consumer Example

Imagine a job charge_user with payload:

json

{
  "user_id": 10,
  "order_id": 555,
  "amount": 49.99
}

If the worker runs this twice, you do not want to charge the user twice.

A simple idempotent design:

In the payments table, have a unique constraint on order_id.
When processing the message:

Start a transaction
Try to insert payment with order_id = 555
If insert succeeds, call the payment provider
If insert fails because order_id exists, do not charge again

So, if the same message is delivered twice, the second time the database insert fails due to the unique constraint, and you skip the external call.

Reliability and Acknowledgments

Acknowledgments (ACKs)

ACKs tell the broker that a message has been processed successfully.

If a worker receives a message and then:

Completes it and sends ACK, the broker removes the message.
Crashes or fails before ACK, the broker keeps or requeues the message.

Many systems have a visibility timeout or unacked timeout:

When a worker receives a message, that message becomes invisible to others for a limited time.
If the worker does not ACK within that time, the message becomes visible again to be redelivered.

Failure Example

Worker receives message M to send an email.
Worker starts sending but the mail server does not respond.
After some time the worker process crashes.
Broker's timeout ends, M is visible again.
Another worker receives M and tries again.

If the email might be sent twice, the worker must handle this scenario carefully. Often sending an email twice is acceptable, but charging money twice is not.

Dead Letter Queues

Sometimes messages keep failing no matter how many times you retry them. For example:

The payload is invalid.
The database row referenced by the message was deleted.
There is a bug in your code.

You do not want these messages to clog the main queue forever.

A dead letter queue (DLQ) is:

A special queue where "poison messages" are sent after they fail too many times.

Example policy:

Try each message up to 5 times.
If it still fails, send it to email_jobs_dead_letter.
Monitor the DLQ and investigate or fix those messages manually.

Important rule: Always define a clear policy for failed messages, for example retry X times, then move to a dead letter queue. Do not allow infinite retries.

Ordering and Parallelism

Message Ordering

Some brokers guarantee that messages in a queue are delivered in the same order they were published. Others do not guarantee global ordering, especially when you use multiple workers.

You must think about:

Do you really need strict ordering?
If you do, can you design the system so that a single worker handles messages for a specific key, such as user ID?

Example:

Messages: "user 1: update profile", "user 1: send welcome email"
If both are independent, ordering does not matter.
If "update profile" must happen before "send welcome email", put that logic in a single job, or serialize processing per user inside the worker.

Parallel Consumers

You can scale a queue by adding more consumers.

Example:

One email worker sends 5 emails per second.
You need to send 1,000 emails quickly.
You can run 10 workers in parallel to reach about 50 emails per second.

The broker distributes messages among consumers. This is a key benefit. You can scale horizontally by starting more worker processes, or more containers.

Using Redis as a Simple Message Queue

Even though there are dedicated message brokers, Redis is often used as a simple queue, especially in Python backends with Celery or RQ.

Redis List as a Queue

Redis has a LIST data structure. You can use:

LPUSH queue_name message to push a message to the left.
BRPOP queue_name to block and pop a message from the right.

Together, they form a FIFO queue: First In, First Out.

Example (conceptual):

Producer:

Serialize message to JSON string.
LPUSH email_queue "<json>".

Consumer:

Call BRPOP email_queue to wait for messages.
When a message arrives, parse JSON and process.

This is simple but lacks some advanced features like built-in retries or dead letter queues. Libraries like Celery add those features on top.

Designing Message Payloads

What to Put in a Message

You must decide what to include in the payload:

Identifiers only: For example, {"user_id": 123} and the worker loads more from the database.
Full data: For example, the full user object, {"user_id": 123, "name": "Alice", "email": "alice@example.com"}.

Trade offs:

Strategy	Pros	Cons
Identifiers	Smaller messages, always use latest data	If data changes, job may behave differently
Full data	Snapshot of data at the moment of enqueue	Larger messages, harder to change schema

Example trade off:

If you want the email to reflect exactly the state of the user at the time of registration, include the important fields in the message.
If you always want the latest user info, send only user_id and look up the rest when processing the job.

Important rule: Avoid putting very large payloads or files directly into messages. Store large data in an external storage (database, file store, S3) and put only references (IDs, URLs) in the message.

Idempotent Design Patterns

Because messages may be processed more than once, you need some concrete patterns.

Use Unique Constraints

Use the database to protect against duplicates:

Add a unique index on a logical key, like order_id or email_verification_token.
The worker tries to insert a new row.
If the insert fails because the row already exists, treat it as "already processed".

Use a "Processed Messages" Table

You can track processed messages explicitly.

Table: processed_messages

Column	Type
`message_id`	UUID
`processed_at`	timestamp

Process flow:

Start a database transaction.
Check if message_id exists in processed_messages.
If it exists, skip processing.
If not, perform the business logic.
Insert message_id into processed_messages.
Commit transaction.

If the same message is delivered again, step 2 will see that it was already handled.

Use Idempotency Keys

Sometimes the producer creates an idempotency key based on the logical action, for example:

Key: "charge:user_id=10:order_id=555"

The worker stores the key and result in the database or cache. If the same key is processed again, it returns the stored result without performing the action again.

Handling Retries

Retries are central to message queues, because many failures are temporary.

When to Retry

Good candidates for retry:

Network errors.
Temporary database failures.
External service timeouts.
Rate limit responses (e.g. HTTP 429).

Bad candidates:

Validation errors.
Permanent "not found" errors caused by wrong data.

Exponential Backoff

Instead of retrying immediately many times, you should wait increasingly longer:

1st retry: after 1 second
2nd retry: after 2 seconds
3rd retry: after 4 seconds
4th retry: after 8 seconds
...

General formula: for retry number $n$ (starting from 1),

$$
\text{delay}_n = \text{base\_delay} \times 2^{(n-1)}
$$

For example, base delay 1 second:

$n = 1$: $1 \times 2^{0} = 1$ second
$n = 2$: $1 \times 2^{1} = 2$ seconds
$n = 3$: $1 \times 2^{2} = 4$ seconds

Important rule: Use exponential backoff with a maximum retry count to prevent hammering a failing service and to keep your system stable.

Most background job libraries let you configure retry strategies so you do not have to implement this math yourself.

Visibility to Users

Using queues changes how your API behaves for clients. You usually deal with eventual completion, not immediate completion.

Common patterns:

1. Return Accepted, Then Poll

POST /generate-report returns { "report_id": 123, "status": "pending" }
Client calls GET /reports/123 every few seconds.
When the worker finishes, status becomes "ready" and the client can download the report.

HTTP status code at submit time is often 202 Accepted, which means, "We accepted your request, but the processing is not finished yet."

2. Webhooks or Push Notifications

Backend enqueues a job for slow processing.
When done, it sends a webhook to the client or pushes a notification via WebSocket.

This is more advanced and requires that the client can receive incoming requests or connections.

Comparing Message Queue Technologies

Since you will often use message queues in Python backends, here is a brief comparison:

Technology	Type	Common usage
Redis	In memory store	Simple queues, short lived jobs, Celery broker
RabbitMQ	Broker	Complex routing, reliable queues, enterprise use
AWS SQS	Cloud queue	Serverless, managed queue in AWS
Kafka	Stream platform	High volume event streams, analytics

As a beginner:

For simple background jobs with FastAPI and Python, Redis + Celery or Redis + RQ is a common and practical starting point.
For large distributed systems where events are important, Kafka or a cloud specific service may be used.

Practical Design Example

Imagine an e commerce backend, you want to:

Accept orders quickly.
Process payment.
Reserve inventory.
Send confirmation email.
Generate invoice PDF.

You could design your flow with message queues:

User calls POST /orders.
API:

Validates input.
Stores order with status "pending".
Enqueues message {"type": "process_order", "order_id": 777}.
Returns 201 Created with the order ID.

Worker order_processor:

Receives process_order message.
Charges the user.
Updates order status to "paid".
Enqueues two more messages:

{"type": "reserve_inventory", "order_id": 777}
{"type": "send_order_confirmation_email", "order_id": 777}

Worker inventory_worker:

Reserves stock.
Updates order status to "ready_to_ship".

Worker email_worker:

Sends email to the customer.

If any step fails:

The message is retried according to your policy.
After too many failures, it goes to a dead letter queue for manual inspection.

The main API is fast and responsive. The heavy work happens behind the scenes.

Summary

Message queues let you:

Offload slow or unreliable tasks to background workers.
Build more responsive APIs.
Decouple different parts of your system.
Scale work horizontally by running more consumers.
Handle temporary failures with retries.

Key ideas to remember:

Producer publishes messages, consumer processes them, broker manages the queue.
Messages may be delivered more than once, design workers to be idempotent.
Use retries with exponential backoff and dead letter queues for hard failures.
Think carefully about message payloads and ordering requirements.
For Python and FastAPI, Redis based queues and Celery are practical tools built on top of these concepts.

These foundations prepare you for the later chapters where you will implement actual background jobs using queues and workers in real backend projects.