Retry Logic at the Gateway — Transient Failure Handling Strategies
In this tutorial, you'll learn about Retry Gateway. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.
Retry logic at the gateway automatically re-attempts failed requests to backend services, handling transient failures like network timeouts and temporary service unavailability.
What You'll Learn
By the end of this lesson, you will implement retry with exponential backoff and jitter, configure retry budgets, handle non-idempotent requests safely, and integrate retries with circuit breakers.
Why It Matters
Transient failures are inevitable in Distributed Systems. A well-configured retry Strategy improves reliability without overwhelming backend services.
Real-World Use
Durga Antivirus Pro retries failed scan requests at the gateway with exponential backoff, ensuring that temporary network issues do not result in failed scans for users.
Retry Flow
flowchart TD
Request-->Gateway
Gateway-->Backend{Backend Response}
Backend-->|Success|Response
Backend-->|Transient Error|Retry{Should Retry?}
Retry-->|Yes|Wait[Exponential Backoff]
Retry-->|No|Error[Error Response]
Wait-->Gateway
Backend-->|Circuit Open|FailFast[Fail Fast]
Retry with Exponential Backoff
A configurable retry handler with exponential backoff and full jitter.
import time
import random
from typing import Callable, Dict, Optional, Tuple
class GatewayRetryHandler:
def __init__(self, max_retries: int = 3,
base_delay: float = 0.1,
max_delay: float = 5.0,
jitter: bool = True):
self.max_retries = max_retries
self.base_delay = base_delay
self.max_delay = max_delay
self.jitter = jitter
self.retryable_statuses = {
429, 500, 502, 503, 504
}
def should_retry(self, status_code: int) -> bool:
return status_code in self.retryable_statuses
def calculate_delay(self, attempt: int) -> float:
delay = min(
self.base_delay * (2 ** attempt),
self.max_delay
)
if self.jitter:
delay = random.uniform(0, delay)
return delay
def execute(self, request_func: Callable,
*args, **kwargs) -> Tuple[int, Dict]:
last_error = None
for attempt in range(self.max_retries + 1):
try:
status, response = request_func(*args, **kwargs)
if self.should_retry(status) and attempt < self.max_retries:
delay = self.calculate_delay(attempt)
time.sleep(delay)
continue
return status, response
except Exception as e:
last_error = e
if attempt < self.max_retries:
delay = self.calculate_delay(attempt)
time.sleep(delay)
continue
return 503, {"error": str(e)}
return 503, {"error": "Max retries exceeded"}
def mock_backend(method: str, path: str) -> Tuple[int, Dict]:
return 503, {"error": "Service Unavailable"}
retry = GatewayRetryHandler(max_retries=3)
status, response = retry.execute(mock_backend, "GET", "/api/scan")
print(f"Final status: {status}, response: {response}")
Retry Budget Management
A retry budget limits the total retry rate to prevent cascading failures.
import time
from collections import deque
from typing import Tuple
class RetryBudget:
def __init__(self, max_retry_ratio: float = 0.2,
window_seconds: int = 60,
min_requests: int = 10):
self.max_ratio = max_retry_ratio
self.window = window_seconds
self.min_requests = min_requests
self.requests: deque = deque()
self.retries: deque = deque()
def record_request(self, is_retry: bool = False):
now = time.time()
self.requests.append(now)
if is_retry:
self.retries.append(now)
cutoff = now - self.window
while self.requests and self.requests[0] < cutoff:
self.requests.popleft()
while self.retries and self.retries[0] < cutoff:
self.retries.popleft()
def can_retry(self) -> Tuple[bool, float]:
total_requests = len(self.requests)
if total_requests < self.min_requests:
return True, 0.0
retry_count = len(self.retries)
current_ratio = retry_count / total_requests
return current_ratio < self.max_ratio, current_ratio
def reset(self):
self.requests.clear()
self.retries.clear()
budget = RetryBudget(max_retry_ratio=0.2)
for _ in range(50):
budget.record_request(is_retry=False)
for _ in range(5):
budget.record_request(is_retry=True)
can, ratio = budget.can_retry()
print(f"Can retry: {can}, current ratio: {ratio:.2%}")
Idempotency Support
Ensure retries do not cause duplicate side effects for non-idempotent operations.
import hashlib
import json
from typing import Dict, Optional, Set
class IdempotencyChecker:
def __init__(self):
self.processed_ids: Set[str] = set()
def get_idempotency_key(self, request: Dict) -> Optional[str]:
key = request.get("headers", {}).get(
"Idempotency-Key"
)
if not key:
key = request.get("headers", {}).get(
"X-Idempotency-Key"
)
return key
def is_processed(self, idempotency_key: str) -> bool:
return idempotency_key in self.processed_ids
def mark_processed(self, idempotency_key: str):
self.processed_ids.add(idempotency_key)
def should_retry_safely(self, request: Dict
) -> Tuple[bool, Optional[str]]:
method = request.get("method", "GET")
if method in {"GET", "HEAD", "OPTIONS"}:
return True, None
id_key = self.get_idempotency_key(request)
if not id_key:
return False, "Non-idempotent request without idempotency key"
if self.is_processed(id_key):
return True, None
return False, "Non-idempotent request not yet processed"
checker = IdempotencyChecker()
request = {
"method": "POST",
"headers": {"Idempotency-Key": "abc-123"}
}
safe, reason = checker.should_retry_safely(request)
print(f"Safe to retry: {safe}, reason: {reason}")
checker.mark_processed("abc-123")
safe, reason = checker.should_retry_safely(request)
print(f"After processing - safe: {safe}")
Common Mistakes
Mistake 1: Retrying Non-Idempotent Requests
Retrying POST requests without idempotency keys can create duplicate resources. Always require Idempotency-Key headers.
Mistake 2: No Backoff Between Retries
Immediate retries create a thundering herd. Always use exponential backoff with jitter.
Mistake 3: Unlimited Retries
Retrying forever can mask permanent failures. Set a maximum retry count and a total timeout.
Mistake 4: Retrying 4xx Errors
Client errors (4xx) will fail on every retry. Only retry server errors (5xx) and rate limits (429).
Mistake 5: No Circuit Breaker Integration
Retries during a cascading failure make things worse. Use circuit breakers to stop retrying when the backend is down.
Practice Questions
- What status codes should trigger a retry at the gateway?
- How does exponential backoff prevent the thundering herd problem?
- What is the purpose of jitter in retry logic?
- Why do retries need an idempotency key for POST requests?
- What is a retry budget and why is it important?
Challenge
Build a gateway retry middleware that retries 5xx errors up to 3 times with exponential backoff and full jitter, respects Idempotency-Key headers for POST requests, and stops retrying when the circuit breaker is open.
FAQ
Mini Project
Build a gateway retry module that retries failed requests with exponential backoff and full jitter, supports configurable max retries and retryable status codes, integrates with an idempotency key store, respects a retry budget of 20 percent, and logs each retry attempt with the delay used.
What's Next
Learn about Circuit Breaker Gateway for preventing cascading failures, or explore Timeout Gateway for request deadline management.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro