Load balancing distributes incoming work across multiple servers, containers, queues, or services so a system stays responsive and reliable. This cheat sheet covers the main strategies used in web systems, distributed services, and cloud infrastructure. College students need it to compare algorithms, reason about tradeoffs, and connect theory with production architecture.
It is especially useful when studying scalability, fault tolerance, and performance engineering.
The core ideas are to measure load, select a target, and adapt when servers fail or traffic changes. Common policies include round robin, weighted round robin, least connections, random choice, consistent hashing, and latency-aware routing. Important formulas include utilization rho = lambda / mu, response time for an M/M/1 queue R = 1 / (mu - lambda), and weighted traffic share p_i = w_i / sum(w).
Good load balancing also depends on health checks, session affinity, backpressure, and avoiding single points of failure.
Key Facts
- Round robin sends request k to server index k mod n, which is simple but ignores differences in server capacity and current load.
- Weighted round robin assigns each server a traffic share p_i = w_i / sum(w), so stronger servers receive more requests.
- Least connections chooses the server with the smallest active connection count c_i, which works well when requests have uneven durations.
- Randomized load balancing with two choices selects two random servers and sends the request to the less loaded one, often written as choose min(load(a), load(b)).
- Consistent hashing maps both keys and servers onto a hash ring so only about 1 / n of keys move when one of n servers is added or removed.
- Queue utilization is rho = lambda / mu, where lambda is arrival rate and mu is service rate, and stable queues require rho < 1.
- For an M/M/1 queue, average response time is R = 1 / (mu - lambda), so response time grows sharply as lambda approaches mu.
- A load balancer must use health checks and remove unhealthy targets, because routing to failed servers increases errors even if the algorithm is otherwise balanced.
Vocabulary
- Load Balancer
- A component that distributes incoming requests or tasks across multiple backend servers or workers.
- Round Robin
- A load balancing policy that sends requests to servers in a fixed repeating order.
- Weighted Routing
- A routing method that sends a larger fraction of traffic to servers with higher assigned weights.
- Consistent Hashing
- A hashing technique that minimizes key movement when servers are added to or removed from a distributed system.
- Session Affinity
- A policy that keeps requests from the same user or session on the same backend server when state locality matters.
- Health Check
- A repeated test used by a load balancer to decide whether a backend server is available and safe to receive traffic.
Common Mistakes to Avoid
- Using round robin for unequal servers is wrong when machines have different capacities, because weaker servers can become overloaded while stronger servers remain underused.
- Ignoring request duration is wrong for long-lived connections, because equal request counts do not mean equal active workload.
- Forgetting health checks is wrong because a load balancer can keep sending traffic to crashed or degraded servers, causing avoidable failures.
- Using sticky sessions without a fallback plan is risky because one overloaded or failed server can trap many users on a bad target.
- Assuming higher utilization is always better is wrong because as rho approaches 1, queueing delay increases rapidly and response times can become unstable.
Practice Questions
- 1 A weighted load balancer has servers A, B, and C with weights 2, 3, and 5. What fraction of traffic should each server receive?
- 2 A service receives lambda = 80 requests per second and one server can process mu = 100 requests per second. Compute utilization rho and the M/M/1 average response time R.
- 3 Using round robin with 4 servers numbered 0 through 3, which server receives request k = 14 if the rule is server index = k mod n?
- 4 A video streaming service has users with long sessions and servers with changing connection counts. Explain why least connections may be a better strategy than simple round robin.
Understanding Load Balancing Strategies Reference
A routing rule only works as well as the signal it uses. Counting active connections can be misleading when one connection sends a tiny file while another streams video for minutes. Counting requests can be misleading when some requests trigger expensive database work.
CPU use, memory pressure, queue length, error rate, and recent response time each reveal a different kind of stress. In practice, a system often combines signals. It may avoid a server whose CPU is saturated even if it has few connections.
Students should distinguish load measurements from the routing policy that reads them. The policy makes a choice. The measurement tells it what is happening.
Queueing explains why a system can feel fine for a long time, then slow down suddenly. When arrivals nearly match service capacity, even small bursts form a waiting line. Each delayed request stays in the system longer, which can increase the number of active connections and consume more memory.
A timeout may cause a client to retry, adding further arrivals at the worst moment. This feedback loop can turn overload into failure. Capacity planning therefore leaves headroom rather than aiming for full use.
A useful mental model is a supermarket line. Adding enough checkout capacity before a busy period is easier than clearing a line after it has grown.
State changes the choice of strategy. A stateless web request can go to any healthy machine. A logged in session stored only in one server's memory creates session affinity, meaning repeat requests need the same target.
Hashing a user identifier can provide this behavior, but it can make recovery harder when a target disappears. Shared session storage reduces that dependency, though it adds network work and creates another service that must be reliable.
Caches have a similar tradeoff. Keeping requests for the same key near one cache can improve hit rates, while too much affinity can produce hot spots when one key becomes extremely popular.
Health checks need careful design because they are decisions made from incomplete evidence. A simple network check confirms that a process accepts connections, not that its database, storage, or downstream dependencies work. A deeper check can catch more failures, but it must be cheap and should not overload the service during an incident.
Checks should require several failures before removing a target and several successes before restoring it. This avoids rapid switching caused by brief network loss.
Real deployments use more than one balancer or a managed front door, since a single routing machine can otherwise become its own point of failure. They use connection draining during updates so existing work can finish while new work goes elsewhere.