KAHIBARO
Discord Login Register ☰
☰

26.7. Load Balancing

Table of Contents

Why Load Balancing Matters

When your backend grows, a single server often becomes a bottleneck. At some point:

Load balancing solves this by spreading incoming requests across multiple backend instances. Instead of one server doing all the work, several servers share it, and a load balancer sits in front to decide which one handles each request.

You can think of the load balancer as a receptionist in a busy clinic: patients come in through one door, but the receptionist sends each patient to one of several doctors so no single doctor is overwhelmed.

Load balancing is central for:

In this chapter, we focus on these aspects and on the strategies used to distribute load, not on how to configure specific tools like Nginx or cloud load balancers, which are covered elsewhere.

Basic Load Balancing Architecture

A typical backend with load balancing looks like this:

text 
                  +----------------------+
   Clients  --->  |    Load Balancer     |  --->  [ App Server 1 ]
 (browsers,       +----------------------+        [ App Server 2 ]
  mobile apps)          |      |                 [ App Server 3 ]
                        v      v
                   multiple backend
                      instances

Key roles:

Important details:

Key Requirement
For simple, reliable load balancing, application servers should be stateless. Store user state in shared systems (database, Redis, etc.), not in local memory that depends on a specific server.

Horizontal Scaling With Load Balancing

Vertical scaling means "bigger server." Horizontal scaling means "more servers." Load balancing is the enabler for horizontal scaling.

Vertical vs horizontal scaling

ApproachHow it scalesProsCons
Vertical scalingAdd more CPU, RAM to one machineSimple to manage, no code changesHardware limits, single point of failure
Horizontal scalingAdd more instances, same appHigh scalability, high availabilityMore complex infrastructure and design

Load balancing is necessary for horizontal scaling because:

Scaling out and back in

With a load balancer, you can:

In many cloud environments, autoscaling works like this:

  1. A metric crosses a threshold (for example, CPU > 70% for several minutes).
  2. Autoscaling adds a new instance.
  3. The load balancer detects the new healthy instance and starts sending traffic to it.

Later, when load decreases:

  1. CPU stays low for some time.
  2. Autoscaling removes instances one by one.
  3. The load balancer stops sending traffic to instances being terminated.

You need to design your app so it tolerates instances appearing and disappearing at any time.

Common Load Balancing Algorithms

The core job of the load balancer is to decide which backend instance should handle a new request. This decision is made by a load balancing algorithm.

Below are the most common ones. In practice, you often combine an algorithm with health checks and weights (for example, some servers are stronger, so they get more traffic).

1. Round Robin

Round robin cycles through servers in order.

Example with 3 servers: A, B, C

Requests are distributed as:

  1. Request 1 → A
  2. Request 2 → B
  3. Request 3 → C
  4. Request 4 → A
  5. Request 5 → B
  6. Request 6 → C
    ... and so on.

Pros:

Cons:

Weighted round robin

You can assign weights based on server capacity.

Suppose:

Then sequence might be:

Server A gets about $\frac{2}{2+1} = \frac{2}{3}$ of the traffic, server B gets $\frac{1}{3}$.

Important Rule
Use weighted round robin when your servers have different capacities. Otherwise slower or smaller servers can be easily overloaded.

2. Least Connections

The next request goes to the server with the fewest active connections.

Example:

New request goes to Server C.

Pros:

Cons:

Least response time (variation)

Some load balancers use a variation: least response time, which considers:

This sends more requests to servers that answer quickly.

3. IP Hash (or consistent hashing)

The server is chosen by applying a hash function to the client IP address.

Example:

text 
server_index = hash(client_ip) mod N

Where:

Effect:

Pros:

Cons:

Consistent hashing

A more advanced form is consistent hashing, which minimizes how many clients move when servers are added or removed. It is often used in distributed caches (for example, Redis clusters) but can also be used in load balancing.

4. Random

Each request is sent to a random server.

You can make this weighted as well:

Pros:

Cons:

5. Custom / application level strategies

Sometimes the application itself participates in load distribution, for example:

In such cases, a simple algorithm (round robin) at the network level is combined with logic in the app that uses other systems (like a service registry, or database sharding rules).

Application-Level Concerns: Statelessness and Sessions

The balancing strategy interacts strongly with how you manage state.

Stateless vs stateful servers

Load balancing is far easier with stateless servers, because:

Session stickiness (affinity)

Sometimes, you still have stateful behavior, like sessions. In that case, you may need session affinity or sticky sessions.

Sticky sessions mean: the same user is consistently routed to the same backend instance.

Common approaches:

MethodHow it worksProsCons
IP hashMap client IP to a server using a hashSimple, no extra dataBreaks with NAT, clients changing IP
Cookie-basedLoad balancer sets a cookie that encodes the targetMore reliable than IP-basedIf target server dies, session may break
App-session basedApp issues its own session ID that includes server IDFully controlled by applicationMore complex, can become brittle

In practice, stateless sessions are preferred for scalable APIs:

Best Practice
Design your backend so that any request can be processed by any instance. Avoid server-local sessions. Use shared storage (database, Redis, etc.) and stateless tokens.

Health Checks and Failover

A critical part of a real load balancing setup is health checking.

What is a health check?

A health check is a periodic request to each backend instance to verify that it is healthy.

Typical patterns:

If a server fails health checks:

Simple vs deep health checks

You can also have multiple levels:

Load balancers often use a readiness endpoint to avoid sending traffic to instances that are starting up, migrating, or overloaded.

Failover

When an instance fails:

  1. Health checks fail repeatedly.
  2. Load balancer marks it as unhealthy.
  3. Traffic is automatically redistributed to other instances.

If all instances are down, the load balancer might:

Your job as a backend engineer is to:

Layer 4 vs Layer 7 Load Balancing (Conceptual)

Without diving deep into networking chapters, it helps to understand that load balancers can operate at different layers.

Layer 7 features you will commonly use:

These options influence how you design your backend routes and endpoints.

Load Balancing and Performance Considerations

Load balancing itself has performance implications.

Overhead and single point of failure

You are adding an extra hop:

text 
Client → Load Balancer → Backend

This introduces:

To mitigate:

Connection pooling and keep-alive

Modern load balancers:

Benefits:

As a backend engineer, you should:

Balancing CPU-bound and I/O-bound workloads

Load balancing does not eliminate internal performance issues:

You often combine:

Load Balancing and Microservices

In a microservices architecture, load balancing happens more than once.

Examples:

  1. Edge / API gateway layer:
    • Clients access a single public endpoint.
    • The gateway or reverse proxy balances across copies of a gateway service.
  2. Internal service-to-service:
    • One service calls another.
    • It can use:
      • A central load balancer, or
      • Client-side load balancing, where the client library knows multiple instances and chooses one to call.
  3. Database / cache clusters:
    • Requests are distributed across database replicas or cache nodes.
    • Often use separate, database-specific balancing methods.

As a backend engineer, you must:

Practical Design Guidelines

Here are concrete rules you can apply when designing backends that will sit behind a load balancer.

1. Design for statelessness

2. Provide robust health endpoints

3. Choose algorithms appropriately

4. Make scaling safe

5. Observe and measure

Simple Example Scenarios

To make the concepts concrete, here are small example scenarios you might encounter.

Scenario 1: Sudden traffic spike

What happens with a proper setup:

Without load balancing and multiple instances:

Scenario 2: One bad instance

You have 4 instances:

With health checks and load balancing:

Without a load balancer:

Scenario 3: Sticky sessions with legacy design

Legacy app stores sessions in local memory, not in a shared store.

User logs in:

Solution:

By understanding how load balancing works conceptually and how it interacts with statelessness, sessions, health checks, and scaling, you can design backend services that perform well and remain available, even under high traffic or partial failures.

Views: 7

Comments

Please login to add a comment.

Don't have an account? Register now!

© 2025 Kahibaro
Visitors: 3489 / 24h
Impressum / Imprint
Datenschutz / Data Policy