27.9. Distributed APIs

Why Distributed APIs?

Modern backend systems often grow beyond a single application and a single server. You might have:

• Multiple services that each expose their own API
• Multiple data centers or regions
• Multiple versions of the same API running at once

All of this creates a distributed API environment. In simple terms:

A distributed API is an API that is implemented and served by multiple services, components, or locations, but should still look like a single coherent API to clients.

This chapter focuses on the practical problems and patterns that are specific to distributed APIs, not on basic REST or HTTP, which you learned earlier.

Typical examples:

• A mobile app calls https://api.example.com, which routes:
• /auth/... to an authentication service
• /orders/... to an order service
• /products/... to a catalog service
• An e‑commerce platform that runs in both eu-central and us-east regions, but clients use one domain.

The main challenges are:

• How to route, aggregate, and transform requests across services
• How to keep APIs consistent and evolvable
• How to handle failures and latency in a distributed environment

Types of Distributed API Setups

1. Single API, Multiple Internal Services

From the client side, there is a single API, for example:

https://api.example.com

Internally, multiple services handle different parts:

A reverse proxy or API gateway forwards requests to the right service.

Characteristics:

• Central entry point for all clients
• Each service can be deployed independently
• Clients see a unified API, even though it is implemented by many services

2. Multiple APIs, Single Client Application

Sometimes one client (a web app or mobile app) calls several APIs directly:

• https://auth.example.com
• https://users.example.com
• https://orders.example.com

Characteristics:

• Less central control
• The client must know about multiple base URLs
• CORS configuration becomes important for browser apps

This is simpler on the backend, but pushes complexity to the client.

3. Multi‑Region APIs

A company might run the same API in different regions:

• https://api-eu.example.com (Europe)
• https://api-us.example.com (USA)

Or use one global domain with smart routing (using DNS or an API gateway) to the nearest region.

Challenges:

• Keeping data consistent across regions
• Handling user moves between regions
• Dealing with regional regulations (for example data residency)

4. Internal Distributed APIs

Within your company, many internal systems talk to each other via APIs, even if there is no public API. For instance:

• Backend A calls Backend B via internal REST calls
• Service C pushes events that other internal services consume

These are distributed APIs too, just not exposed to external users. They still face the same challenges: versioning, contracts, reliability, and so on.

API Gateway and Aggregation

What an API Gateway Does

An API gateway sits in front of multiple services and provides a unified entry point.

Common responsibilities:

• Routing: GET /users/1 goes to user-service, POST /orders goes to order-service
• Authentication and authorization: Validate JWTs or API keys once, not in every service
• Rate limiting: Protect services from too many calls
• Request / response transformation: For example, map legacy fields to new ones
• Logging and metrics: Central place to log and measure API usage

Examples of gateways / reverse proxies:

• Nginx, Traefik, Kong, API Gateway services in cloud providers, or custom FastAPI gateways.

Table of typical concerns:

Aggregation: One Request, Many Services

Some API operations need data from multiple services. For example, an endpoint:

GET /me

might need:

• User data from the User service
• User settings from the Settings service
• Last 5 orders from the Order service

You have 2 main options.

1. Aggregation in the Client

The client calls multiple endpoints:

1. GET /users/me
2. GET /users/me/settings
3. GET /orders?user_id=me&limit=5

Pros:

• Backend stays simpler
• Each service remains independent

Cons:

• More HTTP calls from client
• Difficult to keep a consistent experience if some calls fail
• More complexity in frontend code

2. Aggregation in a Backend Gateway or “BFF”

A Backend For Frontend (BFF) is a service that aggregates data for a specific client type.

Example BFF endpoint:

GET /me

BFF code (pseudocode / Python style):

async def get_me(user_id: int):
    user = await http_get(f"http://user-service/users/{user_id}")
    settings = await http_get(f"http://settings-service/users/{user_id}/settings")
    orders = await http_get(
        f"http://order-service/orders?user_id={user_id}&limit=5"
    )
    return {
        "user": user,
        "settings": settings,
        "recent_orders": orders,
    }

Pros:

• Client uses a single endpoint
• Frontend simpler, fewer network round trips
• You can optimize responses for different clients (web, mobile)

Cons:

• BFF can become complex and large
• Needs careful handling of timeouts and partial failures

Consistency and Data Ownership

Distributed APIs usually work on distributed data. The hard question is: who owns what?

Clear Data Ownership

In a distributed system, each service should have clear ownership of some data.

Example:

Rule of thumb: Only the owner service can change its data. Other services must call the owner via an API or read replicas.

This prevents:

• Conflicting updates
• Data schemas that diverge silently across databases

Consistency Models in Distributed APIs

Two important ideas:

• Strong consistency: After a write, all reads see the new value immediately.
• Eventual consistency: After a write, some reads may see an old value for a while, but eventually all see the new value.

In a single database, strong consistency is easier. In a distributed system, you often accept eventual consistency.

Examples of eventual consistency in APIs:

• After placing an order, /orders/{id} returns it as PENDING, but /me might show no orders for a short time while caches or read models update.
• After updating a username, some parts of the UI show the old username for a few seconds.

Example: User Email Change Flow

Imagine this flow:

1. User calls PATCH /users/me with { "email": "new@example.com" }.
2. User service updates the user in its database.
3. It publishes an event user_email_changed to a message queue.
4. Other services (Billing, Notifications, etc.) listen and update their own data.

Consequences:

• Immediately after step 1, some services might still use the old email.
• Over time, all services will converge to the new email.

Your API contracts and documentation must make this clear: some derived or cached data may lag behind.

Distributed Transactions and Sagas

The Problem with Transactions Across Services

A normal database transaction ensures ACID properties inside one database. In distributed APIs, you might have:

• User service using PostgreSQL
• Order service using PostgreSQL
• Payment service using some external payment gateway

A classic relational transaction cannot span all of these easily.

Example business action:

“Place an order” needs:
1. Create order record
2. Reserve stock
3. Charge payment

If any step fails, you want to roll back. But those steps happen in different systems.

Two‑Phase Commit vs Sagas

Distributed systems sometimes use two‑phase commit (2PC) across databases, but it is complex and can hurt availability, so it is less common in modern microservices.

Instead, many teams use the Saga pattern.

Saga Pattern Overview

A saga is a sequence of local transactions, each with a compensating action that can undo it.

Basic idea:

1. Service A does step 1 and commits locally
2. Service B does step 2 and commits locally
3. Service C does step 3 and commits locally

If step 3 fails, you do compensating actions:

• Compensate step 2
• Compensate step 1

For our order example:

Two styles:

• Orchestration: A central saga orchestrator calls each service in order.
• Choreography: Each service reacts to events and publishes new events.

Simple Saga Example (Choreography)

1. Client calls POST /orders on Order service.
2. Order service creates order PENDING, publishes order_created.
3. Stock service receives order_created, tries to reserve stock:
• If success, publishes stock_reserved.
• If failure, publishes stock_reservation_failed.
4. Payment service receives stock_reserved, tries to charge card:
• If success, publishes payment_succeeded.
• If failure, publishes payment_failed.
5. Order service listens:
• On payment_succeeded, sets order CONFIRMED.
• On stock_reservation_failed or payment_failed, sets order CANCELLED and triggers compensations.

Your public REST API for the client is simple:

POST /orders
GET /orders/{id}

But internally, the system uses events and compensations. The API is just the entry point to a long‑running, distributed process.

What This Means for API Design

A saga is not instantaneous. So your API must:

• Avoid pretending that complex operations are always immediate
• Often return a processing state

For example:

POST /orders
201 Created
Location: /orders/123
{
  "id": 123,
  "status": "PENDING"
}

The client can then poll:

GET /orders/123

Until status becomes CONFIRMED or CANCELLED.

This is much more realistic for distributed APIs than pretending everything is done inside one request/response.

Handling Failures, Timeouts, and Retries

In a distributed environment, failures are normal:

• Network hiccups
• Slow downstream services
• Temporary database issues

Your API must be designed to remain usable and predictable.

Timeouts and Circuit Breakers

If your service calls another service, you must set a timeout. Otherwise your API endpoint can hang and block resources.

Example in Python (simplified):

import httpx
async def call_order_service(order_id: int):
    async with httpx.AsyncClient(timeout=2.0) as client:
        return await client.get(f"http://order-service/orders/{order_id}")

If the call takes more than 2 seconds, it fails.

A circuit breaker adds another layer: if repeated calls fail, it stops sending requests for a while and returns an error immediately, to avoid overloading a failing service.

High‑level behavior of a circuit breaker:

• Closed: Calls flow normally.
• Open: After too many failures, calls fail fast without trying.
• Half‑open: After some time, allow a few test calls to see if the service recovered.

You will usually use libraries or your API gateway to implement this.

Retries and Idempotency

When a call fails, you might retry. But retries can cause duplicate operations, for example double‑charging a payment.

So you must combine:

• Retries: To handle transient errors
• Idempotency: So that repeated identical requests do not create additional side effects

You learned about idempotency before. In distributed APIs it becomes critical.

Example for payments:

POST /payments
Idempotency-Key: 71a4f1...
{ "order_id": 123, "amount": 100.0 }

The server stores that it has processed key 71a4f1... already. If it receives the same key again, it returns the same result without creating a new payment.

Fallbacks and Partial Responses

When aggregating data from multiple services, some downstream calls may fail. You can:

• Return partial data with warnings
• Use cached data as a fallback
• Degrade gracefully, for example hide some section of the UI

Example partial response:

{
  "user": { "id": 1, "name": "Alice" },
  "settings": { "theme": "dark" },
  "recent_orders": null,
  "warnings": ["Recent orders are temporarily unavailable"]
}

Distributed API Versioning and Contracts

Distributed APIs have many services and many clients. You must manage change carefully.

Backward Compatibility First

Because you have many moving parts, it is easier to add than to change or remove.

Principles:

• Add fields, do not change existing meanings
• Do not break existing responses that clients depend on
• Deprecate endpoints slowly and provide a migration plan

Examples of compatible changes:

Versioning Approaches

You learned API versioning before. In a distributed setting:

• Each service might have its own versioning strategy
• The global API gateway may expose its own versions

Common patterns:

• https://api.example.com/v1/users
• Version in header: Accept: application/vnd.example.v1+json

The challenge is synchronizing version changes across many services.

Example: Gateway Version vs Service Version

• Public API gateway exposes /v1/orders/*.
• Internally it calls order-service which has its own internal versions:
• /internal/v1/orders
• /internal/v2/orders

The gateway can gradually route some traffic to /internal/v2/ while keeping public API stable. This is useful for canary releases and backward compatibility.

Contract Testing Between Services

In a distributed system, you should avoid breaking your internal consumers as well.

A contract test validates that a service still returns what its consumers expect.

High level idea:

• Consumer service defines a “contract” for what it expects from provider service’s API.
• Provider runs tests to ensure its implementation satisfies all current contracts.

This is especially helpful when different teams own different services.

Distributed Identity and Authorization

When you have multiple services, you cannot have each one do its own login and identity model independently. You need a consistent identity across your distributed APIs.

Central Authentication, Distributed Authorization

Typical setup:

• One Auth service or Identity Provider (IdP):
• Handles login, logout, tokens
• Issues JWTs or access tokens
• Many resource services:
• Validate tokens
• Apply their own permission rules based on the token claims

Example JWT payload for a distributed system:

{
  "sub": "user-123",
  "email": "alice@example.com",
  "roles": ["user", "premium"],
  "permissions": ["orders:read", "orders:create"],
  "iat": 1692000000,
  "exp": 1692003600
}

All services can read the same sub (user ID) and permissions, and enforce their own authorization logic.

Propagating Identity Across Internal Calls

Suppose the API gateway receives a request with Authorization: Bearer <JWT>, validates it, and routes to the Order service. The Order service then calls the Payment service.

Questions:

• Does Order call Payment with the original user’s identity?
• Or with some internal service identity?

Common patterns:

1. User token propagation
• Order service forwards the user’s token in internal calls.
• Payment service knows who the end user is.
2. Service‑to‑service tokens
• Gateway validates user token and passes only user ID and claims in headers.
• Order calls Payment with a service credential that identifies the Order service.
• Payment service trusts Order and uses the passed user info to do its logic.

Table:

In both cases, your distributed APIs must consistently handle identity and permissions, or you risk security problems.

Observability in Distributed APIs

When an error happens in a distributed API, it can be hard to know where it went wrong. Good observability is essential.

Correlation IDs

A correlation ID is a unique ID attached to a whole request flow.

Example:

1. API gateway receives request, generates X-Request-ID: 5f2c9e...
2. Gateway forwards this header to all downstream services.
3. Each service includes X-Request-ID in logs.

Now you can search logs in all services by this ID and see the path the request took.

Many teams also call this a trace ID when using distributed tracing tools.

Distributed Tracing

Tools like Jaeger, Zipkin, or cloud tracing services allow you to see:

• Timeline of calls between services
• Where most of the time was spent
• Which segment failed

They work by propagating trace headers, for example:

• traceparent
• X-B3-TraceId (in some setups)

From an API design perspective, you must ensure that all services forward tracing headers and do not drop them.

Structured Logging Across Services

In distributed APIs, logs from many services must be combined. You learned about logging already, here the key point is:

• Use structured logs (JSON) with fields like service, request_id, user_id.
• Ensure all services log in a similar way so you can search them centrally.

Example JSON log entry:

{
  "timestamp": "2026-08-28T10:12:34Z",
  "service": "order-service",
  "level": "ERROR",
  "request_id": "5f2c9e...",
  "user_id": "user-123",
  "path": "/orders/123",
  "message": "Payment service timeout"
}

Designing Distributed API Operations

So far, we looked at individual concerns. Let us bring them together in a couple of concrete design patterns.

Pattern 1: Asynchronous Operations with Status Polling

For long‑running actions:

1. Client requests an action.

POST /exports
Content-Type: application/json
{ "type": "orders", "from": "2026-01-01", "to": "2026-01-31" }

2. Server responds:

202 Accepted
Location: /exports/abc123
{
  "id": "abc123",
  "status": "PENDING"
}

3. Client polls:

GET /exports/abc123

4. When completed:

{
  "id": "abc123",
  "status": "COMPLETED",
  "download_url": "https://files.example.com/exports/abc123.csv"
}

Internally, the Export service might:

• Use a message queue and background workers
• Call multiple APIs (Orders, Payments, etc.)
• Store export files in object storage

From the client’s perspective, everything is a simple REST API that acknowledges that work is happening in the background.

Pattern 2: Event‑Driven Updates with Webhooks or WebSockets

In some cases, instead of polling, you push updates:

• Webhooks: Your API calls the client’s URL when something changes.
• WebSockets / SSE: Client listens to a stream of updates.

These fit naturally into distributed architectures, because services already communicate via events.

Example webhook payload:

POST https://client-app.example.com/webhooks/orders
{
  "event": "order.confirmed",
  "order_id": 123,
  "user_id": "user-123"
}

Practical Tips for Beginners

Distributed APIs can feel overwhelming. To approach them step by step:

1. Start with a monolith
   Implement a clean REST API in one application. Separate layers clearly (routing, services, repositories). This makes it easier to split later.
2. Introduce internal services when needed
   For example, split out an auth-service while keeping API gateway simple. Use consistent request / response patterns.
3. Add a small API gateway or reverse proxy
   Use Nginx, Traefik, or a simple FastAPI gateway that routes requests to services. Add authentication and request logging at the gateway.
4. Use IDs and correlation IDs everywhere
• Stable resource IDs, like user_id and order_id
• Correlation IDs for requests
5. Design for retries and idempotency
   Any call that might be retried should be safe to repeat.
6. Accept eventual consistency
   Document which parts of the system may be slightly out‑of‑date and for how long. Design your UX and APIs to handle that.
7. Keep contracts explicit and documented
   Use OpenAPI for each service. Over time you can explore contract testing and stronger schema validation between services.

Summary

Distributed APIs are about making multiple services look like a coherent system:

• A gateway or reverse proxy routes, authenticates, and sometimes aggregates.
• Data ownership and eventual consistency are central design concerns.
• Large multi‑step operations use sagas, not simple single‑database transactions.
• You must handle failures, timeouts, retries, and idempotency carefully.
• Consistent versioning and contracts keep many services and clients compatible.
• Centralized identity, observability, and logging are required to understand and secure your system.

As you build more complex backends, these distributed API patterns will help you evolve from a single app to a robust, multi‑service architecture.

Views: 7