Gateway Scaling — Horizontal and Vertical Scaling Strategies
In this tutorial, you'll learn about Gateway Scaling. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.
Scaling the API gateway ensures it can handle increasing traffic volumes while maintaining low latency and high availability as your user base grows.
What You'll Learn
By the end of this lesson, you will implement horizontal scaling with load balancers, configure auto-scaling policies, manage shared state across instances, and plan multi-region gateway deployments.
Why It Matters
An undersized gateway becomes a bottleneck that limits the entire API platform. Proper scaling ensures your gateway grows with your traffic without becoming a single point of failure.
Real-World Use
Durga Antivirus Pro auto-scales its gateway from 3 to 20 instances during peak hours based on CPU utilization and request rate metrics.
Scaling Decision Flow
flowchart TD
Traffic[Increasing Traffic]-->Decision{Vertical vs Horizontal}
Decision-->|Vertical|Up[Scale Up: More CPU/RAM]
Decision-->|Horizontal|Out[Scale Out: More Instances]
Up-->Limits[Hit Hardware Limits]
Out-->Shared[Need Shared State]
Shared-->Redis[Redis / Database]
Out-->LB[Load Balancer]
LB-->AutoScale[Auto-Scaling Group]
Auto-Scaling Policy
Define auto-scaling rules based on gateway metrics.
from typing import Dict, List, Optional, Tuple
import time
import threading
class AutoScaler:
def __init__(self, min_instances: int = 3,
max_instances: int = 50,
scale_up_threshold: float = 0.7,
scale_down_threshold: float = 0.3,
cooldown: int = 120):
self.min_instances = min_instances
self.max_instances = max_instances
self.current = min_instances
self.scale_up_threshold = scale_up_threshold
self.scale_down_threshold = scale_down_threshold
self.cooldown = cooldown
self.last_scale_action: float = 0
self.metrics_history: List[float] = []
def record_metric(self, utilization: float):
self.metrics_history.append(utilization)
if len(self.metrics_history) > 10:
self.metrics_history.pop(0)
def evaluate(self) -> Optional[str]:
if not self.metrics_history:
return None
now = time.time()
if now - self.last_scale_action < self.cooldown:
return None
avg_util = sum(self.metrics_history) / \
len(self.metrics_history)
if avg_util > self.scale_up_threshold \
and self.current < self.max_instances:
return "scale_up"
if avg_util < self.scale_down_threshold \
and self.current > self.min_instances:
return "scale_down"
return None
def scale_up(self) -> int:
self.current = min(
self.current + 1, self.max_instances
)
self.last_scale_action = time.time()
return self.current
def scale_down(self) -> int:
self.current = max(
self.current - 1, self.min_instances
)
self.last_scale_action = time.time()
return self.current
def get_status(self) -> Dict:
return {
"current": self.current,
"min": self.min_instances,
"max": self.max_instances,
"cooldown_remaining": max(
0, self.cooldown - (
time.time() - self.last_scale_action
)
),
}
scaler = AutoScaler(min_instances=3, max_instances=20)
for util in [0.8, 0.85, 0.82]:
scaler.record_metric(util)
action = scaler.evaluate()
if action == "scale_up":
new_count = scaler.scale_up()
print(f"Scaling up to {new_count} instances")
print(f"Status: {scaler.get_status()}")
Shared State for Scaling
When scaling horizontally, the gateway needs shared state for Rate Limiting, Caching, and sessions.
from typing import Dict, Optional, Any
import json
import time
class SharedStateManager:
def __init__(self, redis_client=None):
self.redis = redis_client
self.local_cache: Dict[str, Any] = {}
self.cache_ttl: Dict[str, float] = {}
def get(self, key: str,
use_local: bool = True) -> Optional[Any]:
if use_local and key in self.local_cache:
if time.time() < self.cache_ttl.get(key, 0):
return self.local_cache[key]
del self.local_cache[key]
if self.redis:
value = self.redis.get(key)
if value:
return json.loads(value)
return None
def set(self, key: str, value: Any,
ttl: int = 60,
sync_to_redis: bool = True):
self.local_cache[key] = value
self.cache_ttl[key] = time.time() + ttl
if sync_to_redis and self.redis:
self.redis.setex(key, ttl,
json.dumps(value))
def increment(self, key: str,
amount: int = 1) -> int:
if self.redis:
return self.redis.incrby(key, amount)
self.local_cache[key] = \
self.local_cache.get(key, 0) + amount
return self.local_cache[key]
def delete(self, key: str):
self.local_cache.pop(key, None)
self.cache_ttl.pop(key, None)
if self.redis:
self.redis.delete(key)
manager = SharedStateManager()
manager.set("config:rate_limit", 100, ttl=300)
limit = manager.get("config:rate_limit")
print(f"Rate limit config: {limit}")
Multi-Region Deployment
Deploy gateways across multiple regions for latency and availability.
from typing import Dict, List, Optional, Tuple
class MultiRegionGateway:
def __init__(self):
self.regions: Dict[str, Dict] = {}
def add_region(self, name: str,
hosts: List[str],
health_endpoint: str,
weight: int = 100):
self.regions[name] = {
"hosts": hosts,
"health": health_endpoint,
"weight": weight,
"healthy": True,
}
def get_closest_region(self,
client_ip: str) -> Optional[str]:
best_region = None
best_latency = float("inf")
for name, config in self.regions.items():
if not config["healthy"]:
continue
region_latency = hash(
f"{client_ip}:{name}"
) % 100
if region_latency < best_latency:
best_latency = region_latency
best_region = name
return best_region
def route_request(self, client_ip: str,
path: str) -> Optional[str]:
region = self.get_closest_region(client_ip)
if not region:
return None
hosts = self.regions[region]["hosts"]
idx = hash(path) % len(hosts)
return hosts[idx]
def mark_unhealthy(self, region: str):
if region in self.regions:
self.regions[region]["healthy"] = False
def mark_healthy(self, region: str):
if region in self.regions:
self.regions[region]["healthy"] = True
mrg = MultiRegionGateway()
mrg.add_region("us-east", ["us-gw1:8080", "us-gw2:8080"],
"/health", weight=100)
mrg.add_region("eu-west", ["eu-gw1:8080", "eu-gw2:8080"],
"/health", weight=80)
host = mrg.route_request("203.0.113.1", "/api/scan")
print(f"Routing to: {host}")
Common Mistakes
Mistake 1: Scaling Without Shared State
Adding instances without shared state breaks rate limiting, caching, and session affinity.
Mistake 2: Vertical Scaling Only
Vertical scaling hits hardware limits. Plan for horizontal scaling from the start.
Mistake 3: No Cooldown Period
Rapid scaling up and down causes thrashing. Use cooldown periods between scaling actions.
Mistake 4: Ignoring Downstream Capacity
Scaling the gateway without scaling backends shifts the bottleneck downstream.
Mistake 5: Single-Region Deployment
A regional outage takes down the gateway. Deploy across at least two regions.
Practice Questions
- What is the difference between vertical and horizontal scaling?
- Why does horizontal scaling require shared state?
- What metrics should trigger auto-scaling?
- How do you handle cross-region latency for gateways?
- What is the maximum number of gateway instances in a cluster?
Challenge
Build an auto-scaling system for the gateway that monitors CPU utilization and request rate, scales up when utilization exceeds 70 percent (max 20 instances), scales down when below 30 percent (min 3), and maintains a cooldown period of 120 seconds between scaling actions.
FAQ
Mini Project
Build a scaling system for the gateway that supports horizontal auto-scaling based on CPU and request rate metrics, shared state via Redis for rate limiting and caching, multi-region deployment with geo-routing, and a cooldown mechanism to prevent scaling thrashing.
What's Next
Learn about Gateway Clustering for high availability architecture, or explore Gateway Kubernetes for container Orchestration.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro