System design deep dive
Load Balancer: deep dive
A load balancer is the traffic cop in front of your servers: it spreads requests across a healthy pool so you can scale out, survive machine failures, and deploy without downtime. This deep dive covers the request path, the main algorithms, and the follow-up questions interviewers actually ask.
How a load balancer works, step by step
- The client resolves
api.example.comthrough DNS and gets the address of the load balancer (often a virtual IP shared by an active and a standby node), not the address of any backend. - The client opens a TCP connection to the load balancer. If the LB terminates TLS, the handshake happens here and the certificate lives on the LB.
- The LB consults its pool of backends and filters out any instance currently marked unhealthy by its health checker.
- It picks one healthy backend using the configured algorithm, for example the server with the fewest open connections.
- An L4 balancer forwards the packets (rewriting addresses); an L7 balancer reads the HTTP request, may add headers such as
X-Forwarded-For, and opens or reuses its own connection to the backend. - The backend responds through the LB (or directly to the client in Direct Server Return setups), and the LB records latency and error metrics for that backend.
- In the background, the health checker probes every backend every few seconds (for example
GET /healthzevery 5s, 3 failures to eject, 2 successes to re-admit).
The important insight is that the load balancer turns a fleet of interchangeable servers into a single logical endpoint. That only works if the servers really are interchangeable, which is why stateless application tiers and externalized sessions go hand in hand with load balancing.
When to use it (and when not to)
- Use it as soon as you run more than one instance of a service: for horizontal scaling, for zero-downtime rolling deploys (drain one instance, upgrade it, re-admit it), and to hide individual machine failures from users.
- Use it internally too. Service-to-service traffic in a large system is balanced either by an internal LB or by client-side balancing inside a service mesh sidecar such as Envoy.
- Skip a dedicated LB for a single-instance prototype, or when a managed platform already balances for you (serverless functions, many PaaS offerings).
- Do not rely on DNS round-robin alone for failover: resolvers and clients cache records well past the TTL, so a dead server keeps receiving traffic for minutes.
Load balancing algorithms compared
| Algorithm | How it decides | Strength | Weakness |
|---|---|---|---|
| Round robin | Next server in a fixed rotation | Trivial, no state | Ignores that some requests are 100x heavier than others |
| Weighted round robin | Rotation biased by a per-server weight | Mixes big and small instance types | Weights are static and drift from reality |
| Least connections | Server with fewest open connections | Adapts to long-lived or slow requests | Needs per-server connection tracking |
| Least response time | Lowest recent latency plus load | Routes around a degraded node | Can oscillate if metrics are noisy |
| Power of two choices | Pick 2 random servers, send to the less loaded | Near-optimal balance with almost no coordination | Slightly less even than a full scan |
| IP / consistent hash | Hash of client IP or a key | Affinity for caches and sessions | Uneven when a few clients dominate traffic |
For hash-based routing to a cache tier, pair the LB with consistent hashing so that adding a server only remaps a small slice of keys instead of nearly all of them.
Making the load balancer itself reliable
A single LB box is a single point of failure, so production setups run at least two. The classic pattern is an active-passive pair sharing a floating IP via VRRP (keepalived): if the active node stops sending heartbeats, the standby claims the IP within a second or two. At larger scale, many LB nodes run active-active behind anycast or DNS, and a cloud LB like AWS ALB is already a distributed fleet you never see. In the interview, one sentence is enough: two or more LBs, health-checked, with automatic failover.
Common mistakes in interviews
- Drawing one LB box and never mentioning how it avoids being a single point of failure.
- Using sticky sessions as the default answer for login state instead of moving sessions to Redis or using stateless JWTs.
- Forgetting health checks, which are what actually delivers the fault tolerance.
- Saying L7 everywhere without acknowledging the extra latency, CPU for TLS, and that long-lived WebSocket connections behave differently from short HTTP requests.
- Ignoring connection draining during deploys, which drops in-flight requests.
Interview shortcut: say which layer (L4 or L7), which algorithm and why, how health checks work, and how the LB itself is made redundant. That covers 90% of what is graded.
Interview questions
›What is the difference between an L4 and an L7 load balancer?
An L4 balancer works on TCP/UDP connections and only sees IPs and ports, so it is extremely fast but cannot inspect requests. An L7 balancer parses HTTP, so it can route /api and /static to different pools, rewrite headers, and terminate TLS, at the cost of more CPU per request.
›How does a load balancer detect a failed server?
Through active health checks (periodic TCP connects or HTTP probes to an endpoint like /healthz) and often passive checks that watch real traffic for errors or timeouts. A server is ejected after a threshold of consecutive failures and re-admitted after a few successes, which avoids flapping.
›Why are sticky sessions considered an anti-pattern?
They pin a user to one server, so load becomes uneven and that user loses their session if the server dies. Scaling in also becomes painful because you cannot remove a server without breaking its users. Storing session data in a shared store keeps every server interchangeable.
›What is connection draining?
When a server is removed for a deploy or scale-in, the LB stops sending it new requests but lets existing ones finish within a timeout, such as 30 seconds. Without it, users see errors on every deploy.
›How would you balance traffic across multiple regions?
Use a global layer, such as GeoDNS or anycast, to send each user to the nearest healthy region, then a regional LB inside each region to spread load across instances. Health information from regions feeds the global layer so a whole region can be drained during an outage.