Load Balancer

A load balancer distributes incoming requests across multiple servers to prevent any single server from becoming overwhelmed. In system design interviews, load balancers appear in virtually every architecture diagram you draw—they're fundamental to achieving scalability, availability, and performance.

This page covers what you need to know for interviews: how load balancers work, the algorithms they use, the difference between L4 and L7 balancing, and when to discuss global versus local load balancing. Understanding these concepts helps you design systems that handle millions of requests without falling over.

Why Load Balancers Matter

Without load balancing, scaling beyond a single server is impossible. Load balancers provide:

• Scalability — Add or remove servers transparently as traffic changes
• Availability — Route around failed servers automatically
• Performance — Direct requests to the least-loaded or fastest server
• Security — Hide internal server topology and provide a single entry point

Global vs Local Load Balancing

Load balancing happens at two levels: routing users to the right data center (global), and routing requests to the right server within a data center (local).

Global Server Load Balancing (GSLB)

GSLB routes users to the nearest or most appropriate data center before requests even reach your infrastructure. It considers:

• Geographic proximity — Route users to the closest data center for lower latency
• Data center health — Failover to another region if one goes down
• Load distribution — Balance traffic across regions during peak times

Implementation approaches:

DNS-based GSLB limitations:

DNS is the simplest form of global load balancing, but it has drawbacks:

• ISPs cache DNS responses, so users from the same ISP get the same IP
• DNS doesn't know about server health in real-time
• DNS packet size limits how many IPs you can return
• Recovery is slow when servers fail (must wait for TTL expiration)

For these reasons, modern systems often use anycast or dedicated GSLB services (AWS Route 53, Cloudflare Load Balancing) rather than relying on DNS alone.

Local Load Balancing

Local load balancers operate within a data center, distributing requests across a pool of servers. This is what most people mean when they say "load balancer."

Local LBs can be placed at multiple points:

• Between clients and web servers (most common)
• Between web servers and application servers
• Between application servers and databases

Layer 4 vs Layer 7 Load Balancers

Load balancers operate at different layers of the network stack, with significant implications for what they can do.

Layer 4 (Transport Layer)

L4 load balancers route based on network information: IP addresses and TCP/UDP ports. They see packets, not requests.

How it works:

1. Client opens TCP connection to load balancer's VIP (Virtual IP)
2. LB selects a backend server
3. LB forwards all packets from that connection to the chosen server
4. Server responses return through the LB (or directly to client via DSR—Direct Server Return, bypassing the LB)

Characteristics:

• Very fast — minimal processing per packet
• Protocol agnostic — works with any TCP/UDP protocol
• Limited routing options — can only use IP/port for decisions
• Connection-based — all packets in a connection go to the same server

Layer 7 (Application Layer)

L7 load balancers understand HTTP/HTTPS. They terminate the client connection, inspect the request, and make a new connection to the backend.

How it works:

1. Client opens connection to LB
2. LB terminates TLS, parses HTTP request
3. LB makes routing decision based on URL, headers, cookies, etc.
4. LB opens new connection to chosen backend
5. LB forwards request and returns response to client

Characteristics:

• Content-aware routing — route by URL path, headers, cookies
• TLS termination — offload encryption from backend servers
• Request manipulation — add/remove headers, rewrite URLs
• Higher latency — more processing per request
• HTTP-specific — only works with HTTP(S) traffic

TLS termination benefits:

• Offload CPU-intensive encryption from backend servers
• Centralized certificate management (update certs in one place)
• Enables traffic inspection and modification (add headers, route by content)
• Connection multiplexing—LB can reuse backend connections across clients

Security consideration: After TLS termination, traffic between the LB and backends is unencrypted. For sensitive data, many organizations use "TLS re-encryption"—the LB terminates the client TLS connection, then opens a new TLS connection to the backend. This adds latency but keeps traffic encrypted end-to-end within your infrastructure.

Comparison

When to Use Each

Use Layer 4:

• Non-HTTP protocols (databases, message queues, game servers)
• Maximum throughput with minimal latency
• First tier in a multi-layer setup (L4 in front of L7)
• Simple round-robin distribution

Use Layer 7:

• HTTP/HTTPS web applications (the common case)
• Routing based on URL paths (e.g., /api to one service, /static to another)
• A/B testing or canary deployments (route by header or cookie)
• Rate limiting per user or API key
• Authentication and authorization at the edge

Load Balancing Algorithms

The algorithm determines how the LB chooses which server handles each request. Different algorithms suit different scenarios.

Static Algorithms

Static algorithms don't consider current server state—they make decisions based on configuration alone.

Round Robin

Requests are distributed sequentially: Server 1, Server 2, Server 3, Server 1, Server 2...

Request 1 → Server A
Request 2 → Server B
Request 3 → Server C
Request 4 → Server A (cycle repeats)

• Simple and predictable
• Works well when servers are identical and requests are similar
• Doesn't account for server load or request complexity

Weighted Round Robin

Like round robin, but servers get requests proportional to their weight. A server with weight 3 gets three times as many requests as one with weight 1.

Weights: A=3, B=2, C=1
Sequence: A, A, A, B, B, C, A, A, A, B, B, C...

• Useful when servers have different capacities
• Still doesn't account for actual load

IP Hash

Hash the client IP to determine the server. Same client always goes to the same server.

server = hash(client_ip) % num_servers

• Provides session affinity without cookies
• Can cause imbalance if some IPs generate more traffic
• Adding/removing servers remaps many clients (see consistent hashing below)

Consistent Hashing

The problem with IP hash is that adding or removing servers remaps most clients. Consistent hashing solves this by mapping both clients and servers to positions on a hash ring—when a server changes, only adjacent clients are affected.

This is crucial for distributed caches and stateful services. See the dedicated Consistent Hashing page for the full explanation, including virtual nodes for load balancing.

Dynamic Algorithms

Dynamic algorithms consider real-time server state, providing better distribution under varying conditions.

Least Connections

Route to the server with the fewest active connections.

• Better than round robin for varying request durations
• Accounts for slow requests that tie up connections
• Requires LB to track connection counts

Weighted Least Connections

Combines weights with connection counting: route to the server with the lowest connections / weight ratio.

Least Response Time

Route to the server with the fastest recent response times.

• Accounts for actual server performance
• Can route around slow servers automatically
• Requires response time tracking

Algorithm Summary

Stateful vs Stateless Load Balancing

An important architectural decision is whether your load balancer maintains state about client sessions.

Stateless Load Balancing

The LB makes independent decisions for each request without remembering previous requests.

How it works:

• Each request is routed based solely on the algorithm (round robin, hash, etc.)
• No session information stored in the LB
• Different requests from the same client may go to different servers

Requirements:

• Backend servers must be stateless, OR
• Session state stored externally (Redis, database)
• Any server can handle any request

Advantages:

• LBs are simple and fast
• Easy to scale LBs horizontally
• LB failure doesn't lose session state

Stateful Load Balancing (Session Affinity)

The LB ensures requests from the same client always go to the same server.

How it works:

• LB maintains a mapping: Client → Server
• All requests from a client go to their assigned server
• Also called "sticky sessions"

Implementation methods:

• Cookie-based — LB injects a cookie identifying the backend server
• IP-based — Hash client IP (less reliable with NAT, proxies)
• Session table — LB maintains explicit client-server mapping

When needed:

• Legacy applications with in-memory sessions
• WebSocket connections (must maintain single connection)
• When external session storage isn't available

Disadvantages:

• Uneven load if some clients generate more traffic
• Server failure loses sessions (users must re-login)
• Harder to scale—can't freely add/remove servers
• LBs become stateful (harder to scale, failover complexity)

Health Checking

Load balancers must detect unhealthy servers and stop sending them traffic.

Health Check Types

Passive (Implicit)

• Monitor response success/failure rates
• If a server returns too many errors, mark it unhealthy
• No extra traffic, but slower detection

Active (Explicit)

• LB periodically sends health check requests
• Dedicated endpoint (e.g., /health or /healthz)
• Faster detection, but adds some load

Health Check Endpoints

A good health endpoint checks that the service is actually working:

// GET /health
{
  "status": "healthy",
  "database": "connected",
  "cache": "connected",
  "uptime": 3600
}

What to check:

• Database connectivity
• Cache connectivity
• Critical external dependencies
• Sufficient disk space, memory

What NOT to do:

• Check things that aren't critical for serving requests
• Make the health check too expensive
• Return unhealthy for transient issues

Handling Failures

When a server fails health checks:

1. LB marks it as unhealthy
2. Traffic routes to remaining healthy servers
3. LB continues checking the failed server
4. When it passes checks again, LB marks it healthy
5. Traffic gradually returns (some LBs "warm up" recovered servers)

Connection Draining

When removing a server (for deploys, scaling down, or maintenance), you don't want to drop active connections mid-request.

Connection draining (also called "deregistration delay") works like this:

1. Mark server as "draining" — stop sending new requests
2. Allow existing connections to complete (with a timeout)
3. After timeout or when all connections close, remove the server

Most cloud LBs handle this automatically with configurable drain timeouts (typically 30-300 seconds). For NGINX, this is achieved through the drain parameter.

Key Metrics to Monitor

Track these metrics to catch issues before they affect users:

• Request rate — Traffic volume, useful for capacity planning
• Error rate — 4xx/5xx responses, indicates application or backend issues
• Latency (p50, p99) — Response time distribution, p99 catches tail latency
• Healthy host count — Number of backends passing health checks
• Connection count — Active connections, helps detect connection leaks

Avoiding Single Points of Failure

A single load balancer is itself a single point of failure. Production systems need redundancy.

Active-Passive (Failover)

Two LBs: one active, one standby. If the active fails, the passive takes over.

• Heartbeat between LBs detects failure
• VIP (Virtual IP) moves to passive LB
• Some brief downtime during failover
• Passive resources underutilized

Active-Active

Multiple LBs all handle traffic simultaneously.

• DNS or anycast routes to multiple LBs
• No wasted capacity
• More complex configuration
• Need to ensure consistent routing

Implementation Options

Software Load Balancers

Modern deployments typically use software load balancers running on commodity hardware or VMs.

Example NGINX configuration:

upstream backend {
    least_conn;                              # Use least connections algorithm
    server app1.example.com:8080 weight=3;   # Higher capacity server
    server app2.example.com:8080 weight=2;
    server app3.example.com:8080 weight=1;
}

server {
    listen 443 ssl;

    location /api/ {
        proxy_pass http://backend;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

This shows weighted least-connections balancing across three servers with URL-based routing.

Cloud Load Balancers

Cloud providers offer managed load balancing services:

Advantages of cloud LBs:

• Fully managed—no servers to maintain
• Auto-scaling built in
• Integrated with other cloud services
• Global options available (GSLB)

Hardware Load Balancers

Physical appliances from vendors like F5 and Citrix. High performance but expensive and inflexible. Mostly legacy—new systems use software or cloud LBs.

Common Interview Patterns

Typical Architecture

Most system designs use this pattern:

Users → CDN → Global LB → Regional LB → App Servers → Database
                              ↓
                            Cache

• CDN handles static content globally
• Global LB routes to the nearest region
• Regional/Local LB distributes across servers within the region

What to Mention in Interviews

When adding load balancers to your design:

1. Placement — "Load balancer in front of our web tier"
2. Type — "L7 since we need URL-based routing"
3. Algorithm — "Least connections for our API servers"
4. Health checks — "Health endpoint checks database connectivity"
5. Redundancy — "Active-passive pair for high availability"

Common Follow-up Questions

Q: How do you handle session data?

"Store sessions in Redis. Servers are stateless, any can handle any request."

Q: What happens if a server goes down?

"Health checks detect the failure within seconds. LB stops routing to that server. Users on that server may need to retry, but they'll hit a healthy server."

Q: How do you scale the load balancer itself?

"For L4, scale horizontally with ECMP (Equal-Cost Multi-Path) routing—the network layer distributes traffic across multiple LBs. For L7, use a cloud managed service that auto-scales, or put an L4 LB in front of multiple L7 LBs."

Q: How do you handle WebSocket connections?

"WebSockets are long-lived TCP connections, so once established they stay bound to a backend. Ensure the LB supports HTTP Upgrade for L7 and has long idle timeouts, or use L4 pass-through for protocol-agnostic handling. Sticky sessions aren't required beyond the connection itself."

Quick Reference

Layer 4 vs Layer 7

Algorithm Selection

Interview Checklist

When discussing load balancers:

• Specified placement in architecture
• Chose L4 or L7 with justification
• Mentioned routing algorithm
• Addressed health checking
• Noted redundancy for availability
• Explained session handling (preferably stateless)
• Connected to horizontal scaling story

What Interviewers Look For

1. Understanding of purpose — You know LBs enable scaling and availability, not just that "you need one."

2. L4 vs L7 awareness — You can explain the difference and choose appropriately. Default to L7 for web apps.

3. Algorithm knowledge — You know round robin, least connections, and when each applies. Don't overcomplicate.

4. Stateless design — You advocate for stateless servers with external session storage, not sticky sessions.

5. Practical thinking — You mention health checks, redundancy, and real products (NGINX, ALB) rather than abstract concepts.

Load balancers are infrastructure—they enable your design but aren't usually the interesting part. Demonstrate competence quickly, then focus on the unique aspects of your system.