Timeout Management at the Gateway — Request Deadlines and Cancellation
In this tutorial, you'll learn about Timeout Gateway. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.
Timeout management at the gateway ensures that slow backend requests do not consume gateway resources indefinitely, protecting both the gateway and downstream services.
What You'll Learn
By the end of this lesson, you will implement per-endpoint timeouts, propagate deadlines to backend services, handle timeout cancellation signals, and configure deadline-aware retry logic.
Why It Matters
Without timeout management, a single slow backend can exhaust gateway connection pools, causing cascading failures that affect all clients.
Real-World Use
Durga Antivirus Pro sets aggressive 5-second timeouts on file scan endpoints at the gateway, with 30-second timeouts on report generation endpoints, preventing slow scans from blocking other requests.
Timeout Flow
flowchart LR
Client-->Gateway
Gateway-->Timer[Start Deadline Timer]
Timer-->Service[Backend Service]
Service-->|Within Timeout|Response
Service-->|Exceeds Timeout|Cancel[Send Cancellation]
Cancel-->TimeoutResponse[504 Gateway Timeout]
Timer-->|Deadline Exceeded|Cancel
Deadline-Aware Gateway Client
A gateway HTTP client that enforces request deadlines and propagates them.
import asyncio
from typing import Dict, Optional, Tuple, Callable
import time
class DeadlineClient:
def __init__(self, default_timeout: float = 30.0):
self.default_timeout = default_timeout
self.endpoint_timeouts: Dict[str, float] = {}
def set_endpoint_timeout(self, path: str, timeout: float):
self.endpoint_timeouts[path] = timeout
def get_timeout(self, path: str) -> float:
for pattern, timeout in self.endpoint_timeouts.items():
if path.startswith(pattern):
return timeout
return self.default_timeout
async def request(self, method: str, path: str,
headers: Optional[Dict] = None,
body: Optional[bytes] = None
) -> Tuple[int, Dict]:
timeout = self.get_timeout(path)
deadline = time.time() + timeout
deadline_header = str(int(deadline * 1000))
request_headers = headers or {}
request_headers["X-Deadline"] = deadline_header
try:
result = await asyncio.wait_for(
self._make_backend_call(
method, path, request_headers, body
),
timeout=timeout
)
return result
except asyncio.TimeoutError:
await self._send_cancellation(
path, deadline_header
)
return 504, {
"error": "gateway_timeout",
"message": f"Backend did not respond within {timeout}s"
}
async def _make_backend_call(self, method, path,
headers, body):
await asyncio.sleep(0.1)
return 200, {"status": "ok"}
async def _send_cancellation(self, path: str,
deadline: str):
print(f"Cancellation sent for {path} (deadline: {deadline})")
async def test_timeout():
client = DeadlineClient(default_timeout=0.05)
status, response = await client.request("GET", "/api/slow")
print(f"Status: {status}, Response: {response}")
asyncio.run(test_timeout())
Per-Endpoint Timeout Configuration
Configure different timeout values for different API endpoints.
from typing import Dict, Optional, Tuple, List
import re
class TimeoutConfig:
def __init__(self):
self.rules: List[Tuple[re.Pattern, float]] = []
def add_rule(self, path_pattern: str, timeout: float):
self.rules.append(
(re.compile(path_pattern), timeout)
)
def get_timeout(self, path: str) -> float:
for pattern, timeout in self.rules:
if pattern.search(path):
return timeout
return 30.0
def get_all_rules(self) -> List[Tuple[str, float]]:
return [
(p.pattern, t) for p, t in self.rules
]
config = TimeoutConfig()
config.add_rule(r"^/api/scan/.*", 5.0)
config.add_rule(r"^/api/reports/.*", 30.0)
config.add_rule(r"^/api/health", 2.0)
endpoints = ["/api/scan/file", "/api/reports/daily", "/api/health", "/api/users"]
for ep in endpoints:
timeout = config.get_timeout(ep)
print(f"{ep}: {timeout}s")
Timeout Propagation with gRPC Deadlines
Propagate gateway timeouts to gRPC backends using gRPC deadlines.
from typing import Dict, Optional, Tuple
import time
class GRPCTimeoutPropagator:
def __init__(self, default_timeout: float = 10.0):
self.default_timeout = default_timeout
def calculate_grpc_deadline(self, path: str,
elapsed: float = 0.0) -> float:
timeout = self.default_timeout
remaining = timeout - elapsed
return max(0.1, remaining)
def add_deadline_to_metadata(self, path: str,
metadata: Dict,
elapsed: float = 0.0) -> Dict:
deadline = self.calculate_grpc_deadline(path, elapsed)
remaining_ms = int(deadline * 1000)
metadata["grpc-timeout"] = f"{remaining_ms}m"
return metadata
def check_deadline(self, start_time: float,
timeout: float) -> Tuple[bool, float]:
elapsed = time.time() - start_time
remaining = timeout - elapsed
if remaining <= 0:
return False, 0.0
return True, remaining
propagator = GRPCTimeoutPropagator(default_timeout=10.0)
start = time.time()
time.sleep(0.5)
ok, remaining = propagator.check_deadline(start, 10.0)
print(f"Deadline OK: {ok}, remaining: {remaining:.1f}s")
metadata = propagator.add_deadline_to_metadata(
"/api/scan/file", {}
)
print(f"gRPC metadata: {metadata}")
Common Mistakes
Mistake 1: Setting Timeouts Too Long
Long timeouts defeat the purpose. Each timeout represents worst-case resource consumption.
Mistake 2: Not Propagating Deadlines to Backends
If the gateway times out but does not tell the backend, the backend continues processing wasted work.
Mistake 3: Hardcoding Timeouts
Timeout requirements change as services evolve. Make timeouts configurable per endpoint.
Mistake 4: Ignoring Connection Timeouts
Setting a read timeout without a connection timeout means the gateway may wait forever to establish a connection.
Mistake 5: Timeout Exceeding Client Patience
If the gateway timeout is longer than the client's own timeout, the client hangs unnecessarily. Keep gateway timeouts tighter.
Practice Questions
- What is the difference between a connection timeout and a request timeout?
- How does deadline propagation help in Distributed Systems?
- Why should timeouts be per-endpoint rather than global?
- What happens to a backend request when the gateway times out?
- How do you determine the right timeout value for an endpoint?
Challenge
Build a timeout management system for the gateway that supports per-endpoint timeouts via configuration, propagates deadlines as X-Deadline headers, sends cancellation requests to backends when timeouts occur, and returns consistent 504 responses with timing details.
FAQ
Mini Project
Build a gateway timeout module that allows per-endpoint timeout configuration, propagates deadlines via the X-Deadline header, cancels backend requests on timeout, and returns consistent 504 Gateway Timeout responses with a Retry-After header.
What's Next
Learn about Circuit Breaker Gateway for handling service failures, or explore Retry Gateway for transient failure recovery strategies.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro