Serverless Circuit Breaker — Resilience Patterns for Lambda and Function-as-a-Service

In this tutorial, you will learn about Serverless Circuit Breaker. We cover key concepts, practical examples, and best practices to help you master this topic.

Serverless circuit breaker patterns handle the unique challenges of function-as-a-service environments: stateless execution, cold starts, ephemeral storage, and concurrent execution limits, using external state stores and coordinated fallback strategies.

flowchart LR
    F[Lambda Function] --> CB{Circuit Breaker}
    CB -->|Check State| Redis[Redis/DynamoDB]
    Redis -->|Closed| External[External Service]
    Redis -->|Open| Fallback[Return Fallback]
    External -->|Fail| Update[Update State in Redis]
    Update -->|Threshold| Store[Store OPEN State]
    CB -->|Half-Open| Probe[Probe Service]
    Probe -->|OK| Reset[Reset to CLOSED]

What You'll Learn

• Serverless circuit breaker state management
• External state stores (Redis, DynamoDB)
• Cold start and concurrent execution
• Lambda circuit breaker patterns
• Distributed state coordination

Why It Matters

Serverless functions are stateless and ephemeral. In-memory circuit breaker state is lost between invocations and across concurrent executions. Circuit breakers must use external state stores and handle the unique concurrency and cold-start characteristics of FaaS.

Real-World Use

DodaTech's Lambda functions use DynamoDB-backed circuit breakers for external API calls. When the payment API fails, the circuit state is stored in DynamoDB with a 30-second TTL. All concurrent Lambda invocations check the same state entry, preventing 1000+ concurrent calls to a failing API.

DynamoDB-Backed Circuit Breaker

import time
import json
import boto3
from decimal import Decimal

class DynamoDBCircuitBreaker:
    def __init__(self, name, table_name, fail_threshold=5, recovery_timeout=30):
        self.name = name
        self.table_name = table_name
        self.fail_threshold = fail_threshold
        self.recovery_timeout = recovery_timeout
        self.dynamodb = boto3.resource('dynamodb')
        self.table = self.dynamodb.Table(table_name)

    def call(self, fn, fallback=None, *args, **kwargs):
        state = self._get_state()

        if state['state'] == 'OPEN':
            if time.time() - state['last_failure'] > self.recovery_timeout:
                self._set_state('HALF_OPEN', time.time())
            else:
                return self._fallback(fallback)

        try:
            result = fn(*args, **kwargs)
            self._set_state('CLOSED', 0, failures=0)
            return result
        except Exception as e:
            failures = state['failures'] + 1
            last_failure = time.time()
            new_state = 'CLOSED'
            if failures >= self.fail_threshold:
                new_state = 'OPEN'
            self._set_state(new_state, last_failure, failures=failures)
            return self._fallback(fallback)

    def _get_state(self):
        try:
            response = self.table.get_item(Key={'name': self.name})
            item = response.get('Item', {})
            return {
                'state': item.get('state', 'CLOSED'),
                'failures': item.get('failures', 0),
                'last_failure': item.get('last_failure', 0),
            }
        except Exception:
            return {'state': 'CLOSED', 'failures': 0, 'last_failure': 0}

    def _set_state(self, state, last_failure, failures=None):
        item = {
            'name': self.name,
            'state': state,
            'last_failure': int(last_failure),
        }
        if failures is not None:
            item['failures'] = failures
        if state == 'OPEN':
            item['ttl'] = int(time.time()) + 3600
        try:
            self.table.put_item(Item=item)
        except Exception:
            pass

    def _fallback(self, fallback):
        if fallback:
            return fallback()
        return None

def call_external_api(data):
    raise ConnectionError("API unavailable")

cb = DynamoDBCircuitBreaker("payment-api", "CircuitBreakers")

for i in range(7):
    result = cb.call(call_external_api, fallback=lambda: "Cached", data=i)
    print(f"Call {i+1}: {result}")
    time.sleep(0.1)

Expected output:

Call 1: Cached
Call 2: Cached
Call 3: Cached
Call 4: Cached
Call 5: Cached
Call 6: Cached (circuit open, from DynamoDB state)
Call 7: Cached (circuit open, from DynamoDB state)

Concurrent Lambda Protection

import time
import threading
import redis

class ConcurrentLambdaCircuitBreaker:
    def __init__(self, name, redis_client, fail_threshold=5, recovery_timeout=30):
        self.name = name
        self.redis = redis_client
        self.fail_threshold = fail_threshold
        self.recovery_timeout = recovery_timeout
        self.prefix = f"cb:{name}"

    def check_and_call(self, fn, fallback=None, *args, **kwargs):
        pipe = self.redis.pipeline()
        pipe.get(f"{self.prefix}:state")
        pipe.get(f"{self.prefix}:failures")
        pipe.get(f"{self.prefix}:last_failure")
        state, failures, last_failure = pipe.execute()

        state = state or b'CLOSED'
        failures = int(failures or 0)
        last_failure = float(last_failure or 0)

        if state == b'OPEN':
            if time.time() - last_failure > self.recovery_timeout:
                self.redis.set(f"{self.prefix}:state", 'HALF_OPEN')
            else:
                return self._fallback(fallback)

        try:
            result = fn(*args, **kwargs)
            self.redis.delete(f"{self.prefix}:state",
                             f"{self.prefix}:failures",
                             f"{self.prefix}:last_failure")
            return result
        except Exception as e:
            new_failures = self.redis.incr(f"{self.prefix}:failures")
            self.redis.set(f"{self.prefix}:last_failure", time.time())
            if new_failures >= self.fail_threshold:
                self.redis.set(f"{self.prefix}:state", 'OPEN')
                self.redis.expire(f"{self.prefix}:state", self.recovery_timeout)
            return self._fallback(fallback)

    def _fallback(self, fallback):
        if fallback:
            return fallback()
        return None

r = redis.Redis(host='localhost', port=6379, db=0)
cb = ConcurrentLambdaCircuitBreaker("external-api", r)

def simulate_concurrent_calls():
    def call_with_fail():
        return cb.check_and_call(
            lambda: (_ for _ in ()).throw(Exception("fail")),
            fallback=lambda: "FALLBACK"
        )

    threads = []
    for i in range(10):
        t = threading.Thread(target=lambda i=i: print(f"Thread {i}: {call_with_fail()}"))
        threads.append(t)
        t.start()

    for t in threads:
        t.join()

simulate_concurrent_calls()
print(f"Final state: {r.get('cb:external-api:state')}")

Expected output:

Thread 0: FALLBACK
Thread 1: FALLBACK
...
Thread 9: FALLBACK
Final state: b'OPEN'

Common Mistakes

• In-memory state in Lambda -- Lambda instances are reused but state is not shared across instances. In-memory circuit breaker state only protects within a single warm instance. Use DynamoDB, Redis (ElastiCache), or Parameter Store for shared state.
• Thundering herd on state check -- 100 concurrent Lambda invocations all checking circuit breaker state simultaneously can overwhelm the state store. Use conditional writes (DynamoDB ConditionExpression) or Redis atomic operations.
• Not handling cold starts -- cold starts have no cached state. The circuit breaker must query the external store on every invocation. Cache the state locally for 1-5 seconds to reduce store read load during warm invocations.
• TTL-based recovery without probes -- setting a fixed TTL for open state causes thundering herd recovery. All concurrent invocations see the state expire at the same time and probe simultaneously. Use probabilistic expiration.
• Ignoring concurrent invocation limits -- Lambda has a burst concurrency limit. If circuit breaker state checks happen before Lambda invocation, you can reject excess invocations at the circuit breaker level before they consume Lambda concurrency.

Practice Questions

1. Why is in-memory circuit breaker state insufficient for Lambda?
2. How does DynamoDB provide shared circuit breaker state across Lambda instances?
3. How do you prevent thundering herd when checking circuit breaker state in Lambda?
4. What is the impact of cold starts on circuit breaker performance?
5. How do TTL-based recovery strategies differ for serverless vs serverful environments?

Challenge

Build a serverless circuit breaker framework: (1) DynamoDB-backed circuit breaker state with TTL for automatic cleanup, (2) Redis (ElastiCache) backed state for lower latency, (3) Lambda layer that provides circuit breaker utility functions, (4) local Caching of circuit state with 1-second TTL to reduce store reads, (5) conditional write pattern to prevent thundering herd on state updates, (6) CWL (CloudWatch Logs) metrics for circuit breaker state, failure count, and fallback rate, (7) SAM/CloudFormation template that deploys the circuit breaker infrastructure, (8) step function integration: circuit breaker state influences step function branching.

FAQ

Mini Project

Build a serverless resilience framework: (1) DynamoDB-backed circuit breaker with atomic counters for concurrent safe failure tracking, (2) Redis-backed circuit breaker for lower-latency scenarios, (3) Lambda layer providing circuit breaker utility functions (check, call, fallback), (4) local state cache with configurable TTL (default 2 seconds), (5) CloudWatch metrics integration: circuit state, failure count, fallback count, state transition count, (6) X-Ray Tracing integration: add circuit breaker state as annotation to trace segments, (7) SAM template for deployment with DynamoDB table and Lambda functions, (8) example Lambda function that calls external API with circuit breaker protection and fallback.

What's Next

Continue with Edge Computing to learn edge circuit breaker patterns. Then explore IoT Patterns for IoT circuit breaker patterns.