Load Balancing
Spreading traffic across machines, and what breaks the moment there's more than one.
35 minDifficulty 3/5infra · scalingAI-writtenWritten by a model on 16 August 2026 and not yet read by a person. Checked automatically: schema, the pedagogical rules the hand-written material is held to, and every diagram parsed for real.
Before this
Why this exists
One server handles 10,000 requests per second until it can't, and adding a second identical server doesn't help unless something actually decides WHICH server each incoming request goes to. A load balancer is that something — the single entry point clients talk to, which distributes traffic across a pool of servers behind it.
The mental model
Think of a load balancer as the host at a restaurant with several open tables — customers don't pick their own table, the host directs each one somewhere, using some rule (the next available table, the shortest line, whichever server is currently least busy) so that no single table gets overwhelmed while others sit empty.
How it works
Round robin is simple but ignores actual server load
Round robin sends requests to servers in a fixed rotating order — server 1, then 2, then 3, then back to 1 — regardless of how busy each one currently is. This works fine when every request costs roughly the same amount of work, but breaks down when some requests are much heavier than others, since a server that happened to get several heavy requests in a row stays overloaded while others sit idle.
Least-connections routing adapts to actual current load
Instead of a fixed rotation, this strategy sends each new request to whichever server currently has the FEWEST active connections — a proxy for how busy it actually is right now. This self-corrects for uneven request costs in a way round robin structurally cannot, at the cost of the load balancer needing to track each server's current connection count.
Health checks remove a failing server from rotation automatically
A load balancer periodically pings each backend server (a lightweight `/health` endpoint, typically) and stops routing traffic to any that fail to respond correctly — a crashed or overloaded server is automatically taken out of rotation without a human needing to intervene, and put back once it starts passing health checks again.
Sticky sessions route a client's requests to the same server, which conflicts with the stateless ideal
If a server holds session state in its own local memory (a user's shopping cart, say), that user's SUBSEQUENT requests must reach the SAME server, or the state appears to vanish — this is a sticky session, and it works, but it undermines load balancing's ability to freely redistribute traffic and complicates scaling. The better long-term fix is making servers stateless (storing session data externally, e.g. in Redis) so any server can handle any request — see `horizontal-scaling`.
The mechanism
Every client connects to the load balancer's single address, never directly to a backend server. The load balancer maintains a pool of known servers, continuously checking their health, and routes each incoming request to one of the currently-healthy servers according to its configured strategy — round robin, least connections, or another algorithm — distributing load without any client needing to know how many servers exist or which one handled their request.
flowchart TD C1[Client] --> LB[Load Balancer] C2[Client] --> LB C3[Client] --> LB LB -->|healthy, least busy| S1[Server 1] LB -->|healthy| S2[Server 2] LB -.->|failed health check| S3[Server 3 - removed]
What people get wrong
- Adding more servers automatically distributes load without any load balancer involved.
- Without something actively routing requests, clients would need to somehow know about and choose between multiple server addresses themselves — a load balancer is specifically what makes multiple servers appear as one, and it's what does the actual distribution. This misconception undersells why load balancers exist at all — 'more servers' alone doesn't solve the distribution problem unless something is directing traffic across them.
- Round robin is always a fine load balancing strategy since it distributes requests evenly by count.
- Round robin distributes requests evenly by COUNT, but not necessarily by actual server LOAD — if requests vary significantly in cost, a server can end up overloaded despite receiving 'its fair share' by count, which least-connections or similar load-aware strategies avoid. Choosing round robin without considering request cost variability can produce a system that looks balanced in monitoring (equal request counts) while actually being unevenly loaded in practice.
- Sticky sessions are a permanent, acceptable solution to needing per-user state.
- Sticky sessions work but reduce the load balancer's flexibility (a server that goes down loses all its sessions, and load can't be freely redistributed) — the more scalable long-term fix is making servers stateless by storing session data externally, so sticky routing isn't needed at all. Sticky sessions are a reasonable short-term or legacy accommodation, but treating them as the final answer rather than a stepping stone toward statelessness leaves scaling and resilience limitations unaddressed.
When not to use it
- Backend servers are already stateless, with no per-server session data.
- A load-aware strategy like least-connections, with no need for sticky sessions at all — any server can genuinely handle any request, which is the ideal load balancing can fully exploit.
- Requests need to be routed based on their CONTENT (e.g. `/api/*` to one pool of servers, `/static/*` to another), not just spread evenly across identical servers.
- A layer 7 (application-aware) load balancer or API gateway that can inspect the request path/headers, rather than a simple layer 4 (connection-level) load balancer that only sees IP and port.
Terms
- Round robin
- — A load balancing strategy that distributes requests to servers in a fixed, rotating order, without regard to current load.
- Least connections
- — A load balancing strategy that routes each new request to the server currently handling the fewest active connections.
- Health check
- — A periodic probe the load balancer sends to each backend server to determine if it should remain in the pool of servers eligible to receive traffic.
- Sticky session
- — A load balancing configuration that routes a given client's requests to the same backend server, typically needed when that server holds per-client state locally.
In an interview
Why might round robin load balancing lead to one server being overloaded even though it received the same number of requests as the others?
- round robin distributes by REQUEST COUNT, not by actual computational cost or current load
- if that server happened to receive several unusually expensive requests, it can be overloaded despite an equal request count
- a load-aware strategy like least-connections adapts to actual current load rather than a fixed rotation
Can you recall it?
Why do sticky sessions conflict with the ideal of freely distributing load, and what's the more scalable alternative?