Table of Contents
Understanding Load Balancing
Load balancing is the practice of distributing network or application traffic across multiple servers so that no single server is overwhelmed. It improves availability, performance, and reliability of services that users access over a network.
In the context of network services, load balancing usually sits in front of web servers, application servers, or other backend services, and it decides where each incoming request should go.
Core Goals of Load Balancing
The main purpose of load balancing is to share work among multiple backend resources. By doing this, load balancing aims to achieve several goals.
First, it improves availability. If one server fails or becomes unreachable, the load balancer can stop sending traffic to it and continue to route requests to the remaining healthy servers. To users, the service still appears to be working.
Second, it improves scalability. When the demand on an application grows, more servers can be added behind the load balancer. Clients still connect to a single address, but behind that single entry point, the total capacity has increased.
Third, it improves performance. By distributing incoming requests, each server handles fewer concurrent users and can respond faster. The load balancer can also make intelligent decisions, such as avoiding overloaded servers.
Finally, it simplifies maintenance. Servers can be taken out of rotation for patching or upgrades while the load balancer continues to route traffic to other servers. This allows planned work with minimal user impact.
Basic Load Balancer Behavior
A load balancer usually presents one virtual address to clients. Clients send their requests to this single IP and possibly a single DNS name. The load balancer receives each connection or packet, then selects one of the backend servers according to its configuration.
There are two broad approaches to how this selection and forwarding happen. One is at the transport or session level where the load balancer terminates the client connection and opens a new connection to the selected server. The other is at lower levels where it forwards packets in a way that the client and server still see each other’s addresses.
In all cases, the load balancer must be consistent for each connection. Once a particular client connection is assigned to a backend server, all packets for that connection must continue to go to the same server until the connection ends. The exact mechanisms used depend on the layer and the technology, which belong to other chapters.
Health Checks and Server Pools
A key feature of load balancers is their ability to check whether backend servers are healthy before sending them traffic. Without health checks, the load balancer might continue to direct users to a server that has crashed or that is not responding correctly.
Backend servers are usually grouped into a pool, also known as a farm or cluster. The load balancer is configured with the list of servers in this pool. It periodically probes each server using some health check method. The simplest checks are ICMP echo, often called ping, but these only verify that the host responds at the network level.
More useful health checks target the actual service. For example, an HTTP health check might send an HTTP request to a specific URL and expect a particular status code or response content. If the response is wrong, or too slow, or absent, the load balancer can mark that server as unhealthy. Once a server is considered unhealthy, it is removed from the active rotation until it passes health checks again.
A load balancer should send user traffic only to servers that pass configured health checks. Unhealthy servers must be excluded from the active pool.
Health checks can be tuned with intervals and thresholds. For example, a server might be declared down only after several consecutive failures. Similarly, it might be declared up again only after several consecutive successful checks. This avoids flipping frequently between up and down states due to temporary glitches.
Common Load Balancing Algorithms
When multiple servers are healthy and able to receive traffic, the load balancer must choose which server to use for each request. This selection is guided by a load balancing algorithm. Different algorithms try to optimize for fairness, performance, or specific traffic patterns.
A few widely used algorithms are:
| Algorithm | Basic idea | Typical use case |
|---|---|---|
| Round Robin | Cycle through servers in order, one request per server | Simple, similar servers and workloads |
| Weighted Round Robin | Use round robin, but give more requests to stronger servers | Mixed capacity servers |
| Least Connections | Send new connection to server with the fewest open sessions | Long lived or uneven connection lengths |
| Source IP Hash | Use client IP to choose server consistently | Basic user stickiness without extra state |
Round robin is conceptually simple. If there are three servers, the first request goes to server A, the second to server B, the third to server C, then back to A again. This works well when all servers have equal capacity and the load is uniform.
Weighted round robin extends this by assigning a weight to each server. A server with weight 2 might receive about twice as many requests as a server with weight 1. This lets administrators reflect differences in CPU, memory, or other resources.
Least connections is better suited to applications where some users stay connected for a long time or consume more server resources. By always choosing the server with the fewest current connections, the load balancer tries to keep utilization even.
Source IP hash and similar hashing methods aim to send a given client to the same server consistently. The algorithm calculates a hash from the client IP, or sometimes from multiple fields, and uses this to pick a server. This can provide a simple form of stickiness so that repeated requests from the same client end up on the same backend.
Session Persistence (Stickiness)
Some applications store user specific state in memory on a single server. For instance, a web application might keep a user’s login session or shopping cart only in RAM on the backend. If a user’s next request goes to a different server that does not have that data, the user might appear logged out or might lose the cart.
To handle such applications, load balancers can provide session persistence, which is also called stickiness. Persistence means that once a client’s session is tied to a particular server, future requests from that session go to the same server for a defined period.
There are several ways to implement persistence. One common method is based on cookies, where the load balancer inserts or reads a specific cookie that identifies the chosen backend. Another method uses the client’s IP address. As long as the address does not change, the load balancer sends that client to the same server.
Session persistence has trade offs. It can improve user experience for stateful applications but can lead to uneven load distribution. If many heavy users are pinned to one server, that server might become overloaded even when others are lightly used. For more scalable designs, applications often move state into shared storage or databases so that any server can handle any user, and the load balancer does not need to keep sessions sticky.
Load Balancing and High Availability
Load balancing is closely connected to high availability. By having multiple backend servers and by automatically checking their health, a load balanced service can survive the loss of individual nodes.
However, the load balancer itself becomes a critical component. If there is only one load balancer, it becomes a single point of failure. To avoid this, production environments often deploy multiple load balancers. These can run in active standby or in active active arrangements, coordinated with shared virtual IP addresses and failover mechanisms.
Active standby means only one load balancer handles traffic at a time while the other stands by, ready to take over. Active active uses both load balancers at the same time, distributing traffic between them, while still providing redundancy if one fails.
In larger designs, load balancing can be layered. An external load balancer might distribute traffic across multiple data centers, while internal load balancers in each data center distribute traffic among local servers. Each layer adds complexity but also higher resilience.
Hardware vs Software Load Balancers
Load balancing can be implemented in dedicated hardware appliances or purely in software. Hardware based solutions package specialized software with tuned hardware and often provide high throughput and advanced features. They are common in traditional data centers.
Software load balancers run on general purpose servers or virtual machines. They are flexible and easier to deploy in cloud environments. Modern software load balancers can handle very high traffic levels, especially when scaled horizontally.
Cloud providers also offer managed load balancing services. In these cases, the cloud platform runs and maintains the underlying infrastructure, and the user configures the load balancer behavior through a control panel or API. This hides the operational details but follows the same basic principles described here.
Load Balancing at Different Layers
Load balancing can operate at different protocol layers. At lower layers it usually works with IP addresses and ports, while at higher layers it can inspect application level data. Since other chapters cover layers in detail, this section focuses only on what is specific to load balancing.
When load balancing works primarily with IP addresses and ports, it is often called transport level or network level load balancing. It forwards connections based on destination IP and port without understanding the details of the application payload. This approach is efficient and suitable for many protocols, not only HTTP.
When load balancing understands the application protocol, it can make more advanced decisions. For example, an HTTP aware load balancer can route requests based on URL paths, hostnames, or HTTP headers. This is useful for directing different parts of a website to different server pools, or for implementing features such as blue green deployments and A/B testing.
The choice of layer affects what kinds of rules the administrator can define, and also the complexity and processing cost of the load balancing function.
Basic Configuration Concepts
Although each product uses its own terminology, most load balancers share a few common configuration ideas. There is usually a front side configuration that defines where clients connect, such as a virtual IP address, port numbers, and possibly protocol settings. There is also a back side configuration that describes backend server addresses, ports, and health checks.
The front side is sometimes called a virtual server or listener. It might listen on port 80 for HTTP or port 443 for HTTPS. The back side might define a pool containing multiple IP addresses and ports for the actual servers that run the application.
The load balancer ties these two together by associating the virtual server with a backend pool and by specifying the algorithm and persistence rules. The administrator can then add or remove servers from the pool as needed, change weights for capacity changes, or adjust health checks when the application behavior evolves.
Load Balancing in Modern Architectures
As applications have moved toward microservices and container based deployments, load balancing has become more dynamic. Instead of a fixed set of long lived servers, backends may come and go frequently, created and destroyed by orchestration systems.
In such environments, load balancers often integrate with service discovery. When new service instances appear, they are automatically added to the pool. When instances are removed, they are taken out of rotation without manual intervention. This allows applications to scale automatically in response to demand.
Internally, many platforms also use multiple layers of load balancing. There can be external load balancing for user facing traffic and internal load balancing for service to service communication. While the details vary, the core ideas of distributing traffic, checking health, and providing redundancy remain the same.
Summary
Load balancing is a central technique in network services that distributes client traffic among multiple backend servers. It improves availability by routing around failed nodes, increases scalability by allowing more servers to be added behind a single entry point, and enhances performance by sharing the load. Through server pools, health checks, selection algorithms, and optional session persistence, load balancers present users with a single stable service while managing complexity behind the scenes.