Load Balancing in API Gateway — Algorithms, Sticky Sessions, and Health Checks

In this tutorial, you will learn about Load Balancing in API Gateway. We cover key concepts, practical examples, and best practices to help you master this topic.

Load balancing in an API gateway distributes incoming requests across multiple backend instances to ensure no single server is overwhelmed, improving availability, fault tolerance, and responsiveness.

What You'll Learn

• Common load balancing algorithms: round-robin, least connections, IP hash
• How health checks prevent routing to failed instances
• Sticky sessions and when to use them

Why It Matters

Without load balancing, a single backend failure takes down the entire service. Traffic spikes overwhelm a single instance, causing slow responses and dropped connections. Load balancing spreads the load, isolates failures, and enables seamless scaling.

Real-World Use

Durga Antivirus Pro runs 12 scan service instances behind its gateway. When a spike in scan requests hits, the gateway distributes them across all 12 instances. If two instances fail, the gateway removes them from rotation and continues serving through the remaining ten.

flowchart LR
    Client["Clients"] --> GW["Gateway\nLoad Balancer"]
    GW --> I1["Instance 1"]
    GW --> I2["Instance 2"]
    GW --> I3["Instance 3"]
    GW --> I4["Instance N"]
    style GW fill:#dbeafe,stroke:#2563eb

Round-Robin

Round-robin cycles through the backend list sequentially. It is simple and works well when all backends have equal capacity.

import itertools

class RoundRobinBalancer:
    def __init__(self, backends):
        self.backends = backends
        self.pool = itertools.cycle(backends)

    def next_backend(self):
        return next(self.pool)

balancer = RoundRobinBalancer([
    "http://scan-1:8080",
    "http://scan-2:8080",
    "http://scan-3:8080"
])

for i in range(6):
    backend = balancer.next_backend()
    print(f"Request {i+1} -> {backend}")

Expected output:

Request 1 -> http://scan-1:8080
Request 2 -> http://scan-2:8080
Request 3 -> http://scan-3:8080
Request 4 -> http://scan-1:8080
Request 5 -> http://scan-2:8080
Request 6 -> http://scan-3:8080

Least Connections

Least connections routes to the backend with the fewest active connections. This handles uneven request processing times better than round-robin.

class LeastConnectionsBalancer:
    def __init__(self, backends):
        self.connections = {b: 0 for b in backends}

    def next_backend(self):
        backend = min(self.connections, key=self.connections.get)
        self.connections[backend] += 1
        return backend

    def release(self, backend):
        self.connections[backend] -= 1

IP Hash (Sticky Sessions)

IP hash consistently routes the same client to the same backend. This is useful when backends maintain in-memory session state.

import hashlib

class IPHashBalancer:
    def __init__(self, backends):
        self.backends = backends

    def next_backend(self, client_ip):
        hash_val = int(hashlib.md5(client_ip.encode()).hexdigest(), 16)
        index = hash_val % len(self.backends)
        return self.backends[index]

balancer = IPHashBalancer([
    "http://session-1:8080",
    "http://session-2:8080",
])

print(balancer.next_backend("192.168.1.1"))
print(balancer.next_backend("192.168.1.1"))
print(balancer.next_backend("10.0.0.5"))

Expected output:

http://session-1:8080
http://session-1:8080
http://session-2:8080

The same client IP consistently maps to the same backend.

Health Checks

Health checks remove unhealthy backends from the pool. Active health checks periodically probe backend endpoints.

import time
import requests

class HealthCheckedBalancer:
    def __init__(self, backends, check_path="/health", interval=10):
        self.backends = {b: True for b in backends}
        self.check_path = check_path
        self.interval = interval
        self.last_check = 0

    def check_health(self):
        now = time.time()
        if now - self.last_check < self.interval:
            return
        self.last_check = now
        for backend in self.backends:
            try:
                resp = requests.get(f"{backend}{self.check_path}", timeout=2)
                self.backends[backend] = resp.status_code == 200
            except requests.RequestException:
                self.backends[backend] = False

    def next_backend(self):
        self.check_health()
        healthy = [b for b, ok in self.backends.items() if ok]
        if not healthy:
            raise Exception("No healthy backends")
        return healthy[0]

Common Mistakes

1. No Health Checks

Without health checks, the load balancer continues sending traffic to failed instances, causing errors for users.

2. Unequal Backend Capacity

Round-robin assumes equal capacity. If one instance is twice as powerful, use weighted round-robin instead.

3. Sticky Sessions Without Fallback

If a backend fails, sticky sessions break because the session data is lost. Use distributed sessions or replicate session state.

4. Ignoring Connection Draining

When removing a backend, in-flight requests should complete before the backend is fully removed. Implement graceful shutdown.

5. Single-Point-of-Failure Balancer

The load balancer itself must be highly available. Run multiple gateway instances behind a DNS round-robin or floating IP.

Practice Questions

1. What is the difference between round-robin and least connections load balancing?
2. Why do health checks need a timeout setting?
3. When would you choose IP hash over round-robin?
4. What is connection draining and why is it important?
5. How can you make the load balancer itself highly available?

Answers:

1. Round-robin cycles through backends sequentially; least connections sends to the backend with the fewest active connections, handling variable processing times better.
2. A timeout prevents a slow or hung backend from blocking the health check and delaying failover detection.
3. Choose IP hash when backends maintain local session state and clients need consistent routing to the same backend.
4. Connection draining allows in-flight requests to complete before a backend is removed, preventing disruption to active users.
5. Run multiple gateway nodes behind a DNS round-robin, floating IP, or an external load balancer like AWS ELB.

Challenge: Implement a weighted round-robin balancer where backends have different capacities (weights). Backend A weight=5, Backend B weight=3, Backend C weight=2.

FAQ

Mini Project

Build a Python load balancer that uses weighted round-robin across three backends. Add active health checks every 15 seconds and remove unhealthy backends from rotation. Log each routing decision.

What's Next

Continue with Rate Limiting in API Gateway to protect backends from excessive traffic, or explore Gateway Authentication for securing API access.