Gateway Performance Optimization — Throughput, Latency, and Resource Tuning
In this tutorial, you'll learn about Gateway Performance. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.
API gateway performance directly impacts every API consumer. Optimizing the gateway ensures low latency, high throughput, and efficient resource utilization.
What You'll Learn
By the end of this lesson, you will implement connection pooling, request pipelining, response Caching, TLS optimization, worker thread tuning, and horizontal scaling for gateway performance.
Why It Matters
Every millisecond added by the gateway is added to every API call. A 10ms gateway overhead multiplied by millions of requests per day translates to significant user-facing latency.
Real-World Use
Durga Antivirus Pro reduced gateway latency from 15ms to 3ms by implementing connection pooling, response caching, and TLS 1.3 with session resumption.
Performance Optimization Areas
flowchart TD
Gateway[Gateway Performance]-->Connection[Connection Pooling]
Gateway-->Cache[Response Caching]
Gateway-->TLS[TLS Optimization]
Gateway-->Thread[Worker Tuning]
Gateway-->Scale[Horizontal Scaling]
Connection-->KeepAlive[Keep-Alive]
Connection-->Pool[Pool Size]
Cache-->InMemory[In-Memory Cache]
Cache-->Redis[Redis Cache]
TLS-->Session[Session Resumption]
TLS-->OCSP[OCSP Stapling]
Connection Pool Optimization
Efficient connection management reduces latency and resource usage.
from typing import Dict, Optional, List
import threading
import time
class ConnectionPool:
def __init__(self, target_host: str,
max_connections: int = 100,
min_connections: int = 10,
idle_timeout: float = 30.0):
self.target_host = target_host
self.max_connections = max_connections
self.min_connections = min_connections
self.idle_timeout = idle_timeout
self.pool: List[Dict] = []
self.active_count = 0
self.lock = threading.Lock()
def acquire(self) -> Optional[Dict]:
with self.lock:
now = time.time()
while self.pool:
conn = self.pool.pop()
if now - conn["created"] < self.idle_timeout:
self.active_count += 1
return conn
if self.active_count < self.max_connections:
conn = self._create_connection()
self.active_count += 1
return conn
return None
def release(self, conn: Dict):
with self.lock:
conn["created"] = time.time()
if len(self.pool) < self.max_connections:
self.pool.append(conn)
self.active_count -= 1
def _create_connection(self) -> Dict:
return {
"host": self.target_host,
"created": time.time(),
"id": id({}),
}
def get_stats(self) -> Dict:
with self.lock:
return {
"active": self.active_count,
"idle": len(self.pool),
"max": self.max_connections,
"min": self.min_connections,
}
pool = ConnectionPool("backend-svc:8080",
max_connections=50)
conn = pool.acquire()
if conn:
pool.release(conn)
print(f"Pool stats: {pool.get_stats()}")
Worker Thread Tuning
Optimize worker thread count for the gateway's workload.
from typing import Dict, Optional
import threading
from queue import Queue
import time
class WorkerPool:
def __init__(self, min_workers: int = 4,
max_workers: int = 32,
queue_size: int = 1000):
self.min_workers = min_workers
self.max_workers = max_workers
self.queue = Queue(maxsize=queue_size)
self.workers: list = []
self.active_workers = 0
self._start_workers(min_workers)
def _start_workers(self, count: int):
for _ in range(count):
worker = threading.Thread(
target=self._worker_loop,
daemon=True
)
worker.start()
self.workers.append(worker)
self.active_workers += 1
def _worker_loop(self):
while True:
task = self.queue.get()
if task is None:
break
try:
task["handler"](task["request"])
finally:
self.queue.task_done()
def submit(self, handler, request):
self.queue.put({
"handler": handler,
"request": request
})
def scale_up(self):
if self.active_workers < self.max_workers:
self._start_workers(1)
def scale_down(self):
if self.active_workers > self.min_workers:
self.queue.put(None)
self.active_workers -= 1
def get_stats(self) -> Dict:
return {
"active_workers": self.active_workers,
"queue_size": self.queue.qsize(),
"min": self.min_workers,
"max": self.max_workers,
}
pool = WorkerPool(min_workers=4, max_workers=16)
pool.submit(lambda r: print(f"Processing: {r}"),
{"path": "/api/scan"})
print(f"Worker stats: {pool.get_stats()}")
TLS Session Resumption
Optimize TLS handshake performance with session resumption.
from typing import Dict, Optional
import time
import hashlib
class TLSSessionCache:
def __init__(self, max_sessions: int = 10000,
ttl: int = 300):
self.sessions: Dict[str, Dict] = {}
self.max_sessions = max_sessions
self.ttl = ttl
def get_session(self, session_id: str
) -> Optional[Dict]:
session = self.sessions.get(session_id)
if not session:
return None
if time.time() > session["expires"]:
del self.sessions[session_id]
return None
session["hits"] += 1
return session["data"]
def store_session(self, session_id: str,
session_data: Dict):
if len(self.sessions) >= self.max_sessions:
oldest = min(
self.sessions.keys(),
key=lambda k: self.sessions[k]["created"]
)
del self.sessions[oldest]
self.sessions[session_id] = {
"data": session_data,
"created": time.time(),
"expires": time.time() + self.ttl,
"hits": 0,
}
def compute_session_id(self, client_ip: str,
user_agent: str) -> str:
raw = f"{client_ip}:{user_agent}"
return hashlib.sha256(
raw.encode()
).hexdigest()[:16]
def get_cache_hit_rate(self) -> float:
total = sum(
s["hits"] for s in self.sessions.values()
)
if total == 0:
return 0.0
return total / len(self.sessions)
cache = TLSSessionCache()
sid = cache.compute_session_id("10.0.0.1",
"DodaBrowser/2.0")
cache.store_session(sid, {"cipher": "TLS_AES_128_GCM_SHA256"})
session = cache.get_session(sid)
print(f"Session resumed: {session is not None}")
Common Mistakes
Mistake 1: Too Many Worker Threads
More threads than CPU cores cause context switching overhead. Set workers to 2-4x CPU cores.
Mistake 2: Connection Pool Exhaustion
A pool that is too small causes request queuing. Monitor pool utilization and size appropriately.
Mistake 3: No Response Caching
Without caching, every request hits the backend. Cache even for a few seconds to multiply throughput.
Mistake 4: Synchronous Logging
Synchronous logging blocks the request path. Use async logging or batch writes.
Mistake 5: Ignoring TLS Overhead
TLS handshake adds 1-3 RTT. Use session resumption and OCSP stapling to reduce overhead.
Practice Questions
- What is the impact of connection pooling on gateway latency?
- How do you determine the optimal worker thread count?
- What is TLS session resumption and how does it improve performance?
- How does response caching affect gateway throughput?
- What metrics indicate the gateway is resource-constrained?
Challenge
Build a performance benchmark for the gateway that measures throughput, P50/P95/P99 latency, and connection pool utilization under increasing load from 100 to 1000 concurrent requests, and identifies the bottleneck.
FAQ
Mini Project
Build a performance optimization suite for the gateway that implements connection pooling with configurable min/max sizes, response caching with configurable TTL, TLS session resumption cache, and a worker pool with auto-scaling based on queue depth, then benchmark the improvements.
What's Next
Learn about Gateway Scaling for horizontal scalability, or explore Gateway Monitoring for performance Observability.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro