26.8. Horizontal Scaling

Understanding Horizontal Scaling

Horizontal scaling is about adding more machines to handle more work. Instead of buying one very powerful server, you use several smaller servers that work together.

This is one of the most important ideas in modern backend systems, especially for applications that need to handle millions of users.

Horizontal vs Vertical Scaling

There are two basic ways to scale a backend system.

Approach	What you add	Example
Vertical scaling	More power to one machine	Upgrade from 4 cores / 8 GB RAM to 32 / 128
Horizontal scaling	More machines	Run 10 servers instead of 1

Vertical scaling is simple to manage, but it has limits. At some point you cannot add more CPU or RAM, or it becomes very expensive.

Horizontal scaling has more moving parts, but it can grow much further. You can keep adding more servers as needed, if your architecture allows it.

Key rule: Horizontal scaling works well only if your application is stateless, or at least minimizes state on each server.

When Horizontal Scaling Helps

Horizontal scaling is useful when:

Your application is I/O bound and spends time waiting for network or disk.
You serve many independent client requests.
You can run multiple identical instances of your application.
You want high availability: if one server fails, others still work.

It is less useful when:

You have a single heavy computation that cannot be split across machines.
Your application logic is tightly coupled to local state in memory.
You depend on local files that differ between servers.

Example: API Server

Imagine an HTTP API server that:

Takes a request.
Reads from a database.
Returns a JSON response.

Each request is mostly independent from others. You can easily put 10 identical API servers behind a load balancer. Each server runs the same code and connects to the same database.

If one server crashes, the load balancer can send traffic to the remaining 9.

Load Balancers and Traffic Distribution

Horizontal scaling usually requires a load balancer in front of your servers.

A load balancer:

Receives all client requests.
Chooses which backend instance will handle each request.
Forwards the request to that instance.
Returns the response back to the client.

Common Load Balancing Strategies

Strategy	Idea	When useful
Round robin	Send each new request to the next server in a list	Simple, servers have similar capacity
Random	Pick a random server	Similar to round robin, simple to implement
Least connections	Choose server with fewest active connections	When request duration varies
IP hash	Choose server based on client IP hash	Want the same client to usually hit the same server

You will see load balancers implemented with tools like Nginx, HAProxy, Traefik, or cloud load balancers from AWS, GCP, or Azure.

Example: Simple Round Robin

Suppose you have 3 servers: A, B, C.

Requests arrive like this:

Request 1 → A
Request 2 → B
Request 3 → C
Request 4 → A
Request 5 → B
Request 6 → C

No client needs to know that there are 3 servers. They only talk to the load balancer.

Stateless Applications and Shared State

Horizontal scaling is easiest if your application servers are stateless.

A stateless server:

Does not store user-specific data in process memory between requests.
Can handle any request without knowing what it did before for that user.
Can be replaced or restarted at any moment without losing user data.

Important: If you store user sessions only in server memory, you will have problems with horizontal scaling. Use shared stores like Redis or the database instead.

Bad Example: In-Memory Sessions

Imagine a login system like this:

User logs in to server A.
Server A stores sessions[user_id] = session_id in memory.
The load balancer sends the next request from the user to server B.

Server B has no idea about the session in server A, so the user appears logged out.

Better Example: Shared Session Store

A better design for horizontal scaling:

All application servers connect to a shared session store, for example Redis.
When a user logs in, the server writes the session into Redis.
Any server that handles a request can read the session from Redis.

This way any server can serve any user.

Database Constraints and Scaling Limits

Horizontal scaling often starts by adding more application servers. However, the database can quickly become the bottleneck.

Often the pattern looks like this:

Step 1: One app server and one database.
Step 2: More app servers behind a load balancer.
Step 3: Database becomes overloaded.

To scale further, you will eventually need:

Better database indexing and query optimization.
Read replicas for handling more read traffic.
Caching layers like Redis to reduce database load.
Sharding or partitioning for very large datasets.

Horizontal scaling of app servers is simpler than scaling databases, but both are usually needed in large systems.

Horizontal Scaling in Practice

Let us see how horizontal scaling fits into a typical production setup.

Typical Architecture

A common horizontally scaled architecture looks like this:

Clients (browsers, mobile apps) send HTTP requests.
A load balancer (for example Nginx) receives the requests.
The load balancer forwards requests to one of several application servers.
Application servers read and write data in:

A database (for persistent storage).
A cache such as Redis (for fast lookups, sessions, rate limiting).

You can increase capacity by:

Adding more application servers.
Optionally adding more worker processes for background jobs.
Scaling cache and database as needed.

Example: Scaling a FastAPI Service

Assume you have a FastAPI application running with Uvicorn.

On a single server you might run:

bash

uvicorn app.main:app --host 0.0.0.0 --port 8000

To scale horizontally you could:

Build a Docker image for your app.
Run the same container on 5 different servers.
Put an Nginx or cloud load balancer in front of those 5 servers.

From the user’s point of view it is still one API endpoint, for example:

text

https://api.example.com

Internally, this points to 5 backend instances.

State, Files, and Shared Storage

Horizontal scaling becomes tricky when your application uses local storage.

Problem: Local File Uploads

Bad pattern:

User uploads an image.
Server A saves it in /var/www/uploads/image.jpg.
Next request from the same user goes to Server B.
Server B looks in /var/www/uploads, but does not have the file.

This breaks the user experience.

Better Pattern: Shared or External Storage

Safer patterns for horizontally scaled systems:

Use object storage like S3 or S3 compatible services.
Use a shared network file system, but this is less common for modern web apps.

The basic rule is:

Important: Do not rely on local files on a single server for user data if you plan to scale horizontally. Use shared or external storage instead.

Health Checks and Auto Scaling

In many cloud environments you can combine horizontal scaling with auto scaling.

Key ideas:

Each application instance exposes a health check endpoint, for example /health.
The load balancer calls this endpoint regularly.
If an instance is unhealthy, the load balancer stops sending traffic to it.
Auto scaling rules can start more instances when traffic is high and stop some when traffic is low.

Health checks are usually simple endpoints that check:

The application process is running.
It can reach essential dependencies, for example the database, Redis.

Simple Health Check Example

A minimal FastAPI health check:

python

from fastapi import FastAPI
app = FastAPI()
@app.get("/health")
def health():
    return {"status": "ok"}

Your load balancer uses this to decide if an instance can receive traffic.

Consistency and Sticky Sessions

Sometimes you cannot avoid having per-user state in memory. In that case, load balancers can use sticky sessions.

Sticky sessions (also called session affinity):

Try to always send the same user to the same server.
Often implemented using a cookie that stores which backend served the first request.

Example:

User makes the first request, gets assigned to server B.
Load balancer sets a cookie like server_id=B.
All further requests from that user are routed to server B.

This can work in some cases, but it has problems:

If server B fails, the user loses their session.
It reduces flexibility, since you cannot freely move traffic.

In modern systems, sticky sessions are usually avoided in favor of shared session stores and stateless application instances.

Horizontal Scaling and Microservices

Horizontal scaling is related to architecture choices such as microservices.

In a monolith, you scale the whole application together.
In microservices, you can scale each service separately.