KAHIBARO
Discord Login Register ☰
☰

22.10. Load Balancing

Table of Contents

Why Load Balancing Matters

When your backend grows, a single server often becomes a bottleneck. It might run out of CPU, memory, or network bandwidth, or need maintenance without downtime. Load balancing solves these problems by distributing incoming traffic across multiple backend instances.

Typical goals of load balancing:

Imagine you have three identical FastAPI application servers behind Nginx. Users only see one domain, for example api.example.com. Nginx accepts all incoming requests and forwards each one to one of the three backend servers according to a load balancing strategy.

Key idea: A load balancer sits between clients and your backend instances and decides which backend should handle each incoming request.

You can think of it as a traffic cop at a junction, directing cars to different roads to avoid congestion.

Basic Load Balancing Architecture

A common production architecture looks like this:

Client → Internet → Reverse Proxy / Load Balancer → Application Servers → Database / Cache / Other services

You may have:

The client never talks directly to an application server. It sends all requests to the load balancer, which:

  1. Receives the HTTP or HTTPS request.
  2. Chooses a backend server.
  3. Forwards the request.
  4. Receives the response from the backend.
  5. Sends the response back to the client.

If one backend server goes down, the load balancer detects this through health checks and stops sending requests to that server.

Load Balancing Strategies

Load balancers use different algorithms to decide which backend should handle a request. Knowing the basic strategies helps you choose the right one for your backend.

Below is a quick overview:

StrategyIdeaGood for
Round robinNext server in turnSimple, similar servers
Weighted round robinPrefer stronger serversMixed capacity servers
Least connectionsServer with fewest active connectionsLong-lived or uneven requests
IP hashSame client IP goes to same serverBasic session stickiness
URL / header basedRoute based on path or headerMicroservices, A/B testing, canary releases
RandomRandom serverVery simple, sometimes good enough

Round Robin

Round robin cycles through the list of backends:

  1. Request 1 → Server A
  2. Request 2 → Server B
  3. Request 3 → Server C
  4. Request 4 → Server A
  5. and so on.

This works well when:

Example Nginx configuration using round robin:

nginx 
upstream api_backend {
    server 10.0.0.10:8000;
    server 10.0.0.11:8000;
    server 10.0.0.12:8000;
}
server {
    listen 80;
    server_name api.example.com;
    location / {
        proxy_pass http://api_backend;
    }
}

Nginx default behavior for upstream without extra parameters is round robin.

Weighted Round Robin

Weighted round robin assigns more requests to stronger servers.

For example:

You might give weights 1, 2, 4 respectively. Over time:

Example Nginx configuration:

nginx 
upstream api_backend {
    server 10.0.0.10:8000 weight=1;
    server 10.0.0.11:8000 weight=2;
    server 10.0.0.12:8000 weight=4;
}

This is useful when:

Least Connections

Least connections sends the next request to the backend with the fewest active connections.

This is better than round robin when:

Example with Nginx:

nginx 
upstream api_backend {
    least_conn;
    server 10.0.0.10:8000;
    server 10.0.0.11:8000;
    server 10.0.0.12:8000;
}

If:

Then new requests go to Server C until its connection count grows.

IP Hash (Session Affinity)

Many backend applications use in-memory sessions or cache on each server. If a user hits a different server on each request, their session may disappear. IP hash keeps clients on the same server.

Idea:

Example in Nginx:

nginx 
upstream api_backend {
    ip_hash;
    server 10.0.0.10:8000;
    server 10.0.0.11:8000;
}

All requests from 203.0.113.5 will always go to the same backend server.

This is a form of session stickiness. It is less precise when clients are behind a NAT or proxy, because many users share a single IP.

Other Strategies and Layer 7 Routing

Load balancers at HTTP level can inspect:

You can then route requests to different backends.

Some examples:

Example Nginx configuration:

nginx 
upstream users_service {
    server 10.0.0.10:8000;
}
upstream orders_service {
    server 10.0.0.20:8000;
}
server {
    listen 80;
    server_name api.example.com;
    location /api/users/ {
        proxy_pass http://users_service;
    }
    location /api/orders/ {
        proxy_pass http://orders_service;
    }
}

This is useful in microservice architectures or when doing canary releases.

Health Checks and Failover

A load balancer must know which backends are healthy. If it sends traffic to a dead server, users will see errors or timeouts.

There are two basic types of health checks:

Typical health check behavior:

  1. Every few seconds the load balancer sends a request to each backend:
    • For example GET /health.
  2. If the backend returns an unhealthy status (for example 500) or no response:
    • Mark the backend as down.
    • Stop sending new user requests to it.
  3. After a few successful health checks:
    • Mark it as up again.
    • Slowly start sending traffic again.

Example of a very simple health endpoint in FastAPI:

python 
from fastapi import FastAPI
app = FastAPI()
@app.get("/health")
def health():
    return {"status": "ok"}

In Nginx you usually combine built-in checks and, if needed, modules or an upstream such as a cloud LB that does health checks.

Important rule: A backend that fails health checks should not receive production traffic until it is healthy again.

Failover simply means:

This is a core part of high availability.

Session Stickiness and Stateful Backends

Many beginner backends use stateful techniques:

If the load balancer sends a user to a different backend on each request, the user may:

There are three common solutions.

1. Client-independent Session Store

Best practice is to make your application stateless by storing sessions and cache in shared services, for example:

Then any backend server can serve any request, because all use the same shared state.

High level flow:

  1. User logs in.
  2. Backend creates a session in Redis.
  3. Backend sets a session cookie on the client.
  4. On each request, any server can read the session from Redis.

This removes the need for session stickiness and makes load balancing easier.

2. Sticky Sessions at the Load Balancer

If you cannot change your application, you can use sticky sessions.

Options include:

Example idea:

  1. First request: user hits load balancer.
  2. Load balancer chooses backend B, sets a cookie, for example LB_NODE=B.
  3. Future requests with LB_NODE=B go to backend B.

Some cloud load balancers support application cookies and implement this for you.

3. Keep Sessions on the Client

Another stateless approach is to store session data on the client, for example JWT access tokens for authentication. The backend only needs to verify and interpret the token, no server memory needed.

However, JWT and token-based auth have their own security and design considerations covered in other chapters.

Layer 4 vs Layer 7 Load Balancing

Load balancers can work at different layers of the network stack.

TypeOSI LayerWhat it seesTypical tools
Layer 4 (L4)TransportIP addresses, TCP/UDP portsHAProxy (L4 mode), AWS NLB
Layer 7 (L7)ApplicationHTTP headers, URLs, cookiesNginx, Traefik, AWS ALB

Layer 4 Load Balancing

L4 load balancers do not understand HTTP. They only see TCP connections from clients to the load balancer and forward them to backend servers.

Characteristics:

Example use cases:

Layer 7 Load Balancing

L7 load balancers understand the protocol, for example HTTP or HTTP/2.

They can:

For most web APIs and backends, you will use L7 load balancing, for example Nginx as a reverse proxy.

Example: Load Balancing a FastAPI Application with Nginx

Let us put everything together with a complete example.

Step 1: Run Multiple Application Servers

Assume you have a FastAPI application in app.main:app. You run three Gunicorn + Uvicorn workers on different servers or ports.

For example, on three different machines:

Or on one machine with different ports for demo purposes.

Start Gunicorn + Uvicorn workers on each:

bash 
gunicorn -k uvicorn.workers.UvicornWorker app.main:app -b 0.0.0.0:8000

Step 2: Configure Nginx as a Load Balancer

Nginx config:

nginx 
http {
    upstream fastapi_app {
        least_conn;  # or use round robin by removing this line
        server 10.0.0.10:8000;
        server 10.0.0.11:8000;
        server 10.0.0.12:8000;
    }
    server {
        listen 80;
        server_name api.example.com;
        # Optional: limit max connections per backend connection
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        location / {
            proxy_pass http://fastapi_app;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }
    }
}

Now any request to http://api.example.com is forwarded to one of the three FastAPI backends.

You can test which backend you hit by adding an endpoint that returns the hostname or environment variable unique to that instance.

Example FastAPI endpoint:

python 
import socket
from fastapi import FastAPI
app = FastAPI()
@app.get("/whoami")
def whoami():
    return {"hostname": socket.gethostname()}

Call /whoami several times and you should see different hostnames if the load balancer is distributing traffic.

Scaling and Auto Scaling

Load balancing allows horizontal scaling by adding or removing servers.

Manual Scaling

When you see your CPU or response time increase, you can:

  1. Start a new backend server.
  2. Add it to the load balancer configuration.
  3. Reload the load balancer.

For Nginx, you typically reload without downtime:

bash 
sudo nginx -s reload

If health checks are configured, the new server will start to receive traffic as soon as it is healthy.

Auto Scaling

Cloud platforms often offer auto scaling groups:

Example metrics to watch:

Auto scaling is out of scope to configure in detail here, but you should know that load balancing is a prerequisite for scaling horizontally.

Load Balancing and Sticky Problems

Load balancing introduces some common pitfalls.

Problem 1: Caching on One Node

Imagine you cache frequently used data in memory on each server. One server may have the data in cache, another may not. So users see different performance depending on which server they hit.

Solutions:

Problem 2: Non-Idempotent Requests Retried

If the load balancer or a proxy retries a request, and the request is not idempotent, operations may be repeated. For example POST /orders may create two orders instead of one.

You will learn about idempotency more in advanced API chapters, but with load balancing you must:

Problem 3: Source IP Lost

Some reverse proxies hide the original client IP from backends. Your application may always see the load balancer IP, which breaks rate limiting or logging.

Solution: use X-Forwarded-For or a similar header and configure your application or framework to trust it.

Nginx example:

nginx 
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

Your backend framework usually has settings to use these headers as the real client IP.

Practical Tips for Beginners

To effectively use load balancing in your backend projects:

Rule of thumb: A load balancer does not fix a slow application. It only spreads the load. Always profile and optimize your backend first, then scale and load balance.

With these concepts and examples, you should be able to understand how load balancing fits into your backend architecture and how to configure simple load balancing with tools like Nginx.

Views: 7

Comments

Please login to add a comment.

Don't have an account? Register now!

© 2025 Kahibaro
Visitors: 3517 / 24h
Impressum / Imprint
Datenschutz / Data Policy