gRPC Circuit Breaker — Resilience Patterns for gRPC Microservice Calls

In this tutorial, you will learn about grpc circuit breaker. We cover key concepts, practical examples, and best practices to help you master this topic.

gRPC circuit breaker patterns use interceptors to wrap unary and streaming calls with circuit breaker state management, mapping gRPC status codes to failure counts, protecting long-lived streaming connections, and integrating with service mesh sidecars for centralized resilience.

flowchart LR
    C[gRPC Client] --> I[Circuit Breaker
Interceptor]
    I -->|Unary| Call[Unary Call]
    I -->|Stream| S[Streaming Call]
    Call -->|OK| OK[Return Response]
    Call -->|Status Error| Fail[Increment Failure]
    S -->|Stream Broken| Fail
    Fail -->|Threshold| Open[Open Circuit]
    I -->|Open| FB[Fast Fail / Fallback]

What You'll Learn

• gRPC unary interceptor circuit breaker
• Streaming call protection
• gRPC status code mapping
• Service mesh integration (Istio, Linkerd)
• Bidirectional streaming backpressure

Why It Matters

gRPC's persistent connections and streaming calls require different circuit breaker handling than HTTP. A broken stream may not trigger per-call failures, and bidirectional streams need backpressure-aware protection. Interceptors provide transparent circuit breaking without changing service code.

Real-World Use

DodaTech's gRPC Microservices use a Go circuit breaker interceptor for all cross-service calls. When the product service returns consecutive Unavailable status codes, the circuit opens within 5 failures and returns cached product data from the gRPC interceptor's fallback.

Go gRPC Unary Interceptor

package main

import (
    "fmt"
    "golang.org/x/net/context"
    "google.golang.org/grpc"
    "google.golang.org/grpc/codes"
    "google.golang.org/grpc/status"
    "github.com/sony/gobreaker"
    "time"
)

func CircuitBreakerUnaryInterceptor(cb *gobreaker.CircuitBreaker) grpc.UnaryClientInterceptor {
    return func(ctx context.Context, method string, req, reply interface{},
        cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {

        _, err := cb.Execute(func() (interface{}, error) {
            err := invoker(ctx, method, req, reply, cc, opts...)
            if err != nil {
                st, _ := status.FromError(err)
                switch st.Code() {
                case codes.Unavailable, codes.DeadlineExceeded,
                     codes.Internal, codes.ResourceExhausted:
                    return nil, err
                default:
                    return nil, nil
                }
            }
            return nil, nil
        })

        if err == gobreaker.ErrOpenState {
            return status.Error(codes.Unavailable, "service circuit breaker open")
        }
        return err
    }
}

func main() {
    cb := gobreaker.NewCircuitBreaker(gobreaker.Settings{
        Name:        "product-service",
        MaxRequests: 5,
        Interval:    60 * time.Second,
        Timeout:     30 * time.Second,
        ReadyToTrip: func(counts gobreaker.Counts) bool {
            return counts.TotalFailures >= 5
        },
    })

    interceptor := CircuitBreakerUnaryInterceptor(cb)
    conn, _ := grpc.Dial("localhost:50051",
        grpc.WithUnaryInterceptor(interceptor),
        grpc.WithInsecure())
    defer conn.Close()

    fmt.Println("gRPC client with circuit breaker interceptor")
}

Expected output:

gRPC client with circuit breaker interceptor
[when circuit open] rpc error: code = Unavailable desc = service circuit breaker open

gRPC Streaming Interceptor

package main

import (
    "fmt"
    "sync/atomic"
    "google.golang.org/grpc"
    "google.golang.org/grpc/codes"
    "google.golang.org/grpc/status"
    "github.com/sony/gobreaker"
)

type streamCircuitBreaker struct {
    cb *gobreaker.CircuitBreaker
    grpc.ClientStream
    open int32
}

func (s *streamCircuitBreaker) RecvMsg(m interface{}) error {
    err := s.ClientStream.RecvMsg(m)
    if err != nil {
        st, _ := status.FromError(err)
        if st.Code() == codes.Unavailable {
            atomic.StoreInt32(&s.open, 1)
        }
    }
    return err
}

func StreamingCircuitBreakerInterceptor(cb *gobreaker.CircuitBreaker) grpc.StreamClientInterceptor {
    return func(ctx context.Context, desc *grpc.StreamDesc,
        cc *grpc.ClientConn, method string,
        streamer grpc.Streamer, opts ...grpc.CallOption) (grpc.ClientStream, error) {

        clientStream, err := streamer(ctx, desc, cc, method, opts...)
        if err != nil {
            _, cbErr := cb.Execute(func() (interface{}, error) {
                return nil, err
            })
            if cbErr != nil {
                return nil, status.Error(codes.Unavailable,
                    "circuit breaker open for stream")
            }
        }

        return &streamCircuitBreaker{
            cb:           cb,
            ClientStream: clientStream,
        }, nil
    }
}

func main() {
    cb := gobreaker.NewCircuitBreaker(gobreaker.Settings{
        Name: "inventory-stream",
    })
    fmt.Println("gRPC streaming client with circuit breaker interceptor")
}

gRPC Status Code Mapping

import grpc
from enum import Enum

class CircuitBreakerStatusMapper:
    FAILURE_CODES = {
        grpc.StatusCode.UNAVAILABLE: True,
        grpc.StatusCode.DEADLINE_EXCEEDED: True,
        grpc.StatusCode.INTERNAL: True,
        grpc.StatusCode.RESOURCE_EXHAUSTED: True,
        grpc.StatusCode.CANCELLED: False,
        grpc.StatusCode.INVALID_ARGUMENT: False,
        grpc.StatusCode.NOT_FOUND: False,
        grpc.StatusCode.PERMISSION_DENIED: False,
        grpc.StatusCode.UNAUTHENTICATED: False,
        grpc.StatusCode.UNIMPLEMENTED: False,
        grpc.StatusCode.OUT_OF_RANGE: False,
        grpc.StatusCode.DATA_LOSS: True,
        grpc.StatusCode.FAILED_PRECONDITION: False,
        grpc.StatusCode.ABORTED: False,
        grpc.StatusCode.ALREADY_EXISTS: False,
    }

    @classmethod
    def should_count_as_failure(cls, status_code):
        return cls.FAILURE_CODES.get(status_code, False)

    @classmethod
    def get_fallback_status(cls, code):
        if code in (grpc.StatusCode.UNAVAILABLE, grpc.StatusCode.DEADLINE_EXCEEDED,
                     grpc.StatusCode.INTERNAL):
            return grpc.StatusCode.UNAVAILABLE
        return code

mapper = CircuitBreakerStatusMapper()
codes_to_test = [
    grpc.StatusCode.UNAVAILABLE,
    grpc.StatusCode.INVALID_ARGUMENT,
    grpc.StatusCode.DEADLINE_EXCEEDED,
    grpc.StatusCode.NOT_FOUND,
]

for code in codes_to_test:
    result = mapper.should_count_as_failure(code)
    fallback = mapper.get_fallback_status(code)
    print(f"{code.name}: failure={result}, fallback={fallback.name}")

Expected output:

UNAVAILABLE: failure=True, fallback=UNAVAILABLE
INVALID_ARGUMENT: failure=False, fallback=INVALID_ARGUMENT
DEADLINE_EXCEEDED: failure=True, fallback=UNAVAILABLE
NOT_FOUND: failure=False, fallback=NOT_FOUND

Common Mistakes

• Counting all gRPC errors as failures -- gRPC returns errors for normal conditions (NOT_FOUND, INVALID_ARGUMENT). Only count server-side errors (UNAVAILABLE, INTERNAL, DEADLINE_EXCEEDED, RESOURCE_EXHAUSTED) as circuit breaker failures.
• No streaming call protection -- streaming gRPC calls keep connections open. If a stream breaks due to server failure, subsequent RecvMsg calls fail. Wrap stream RecvMsg with circuit breaker failure counting.
• Timeout shorter than gRPC deadline -- gRPC has its own deadline mechanism. Circuit breaker timeout must be longer than the gRPC deadline. Otherwise, the circuit breaker counts in-flight calls as failures.
• Interceptor per call instead of per service -- creating a circuit breaker per gRPC method leads to many breakers. Group methods by service and use one breaker per service. Each method failure contributes to the same service breaker.
• Not handling connection-level failures -- gRPC channel-level failures (dial errors, connection drops) should also count toward circuit breaker failure thresholds. Wrap the dial and connection creation with circuit breaker protection.

Practice Questions

1. Which gRPC status codes should trigger circuit breaker failures?
2. How does streaming circuit breaker differ from unary circuit breaker?
3. Why should you not count all gRPC errors as failures?
4. How do service mesh sidecars add circuit breakers to gRPC?
5. What is the relationship between gRPC deadline and circuit breaker timeout?

Challenge

Build a gRPC circuit breaker interceptor: (1) interceptor for both unary and streaming calls (client side), (2) status code mapping: only count UNAVAILABLE, INTERNAL, DEADLINE_EXCEEDED, DATA_LOSS as failures, (3) per-service circuit breakers with separate configurations, (4) fallback: return cached response or error with specific status code, (5) streaming protection: wrap RecvMsg to detect stream failures, (6) Prometheus metrics: circuit state per service, failure count by status code, (7) server-side interceptor that rejects requests when the service is overloaded (server-side circuit breaker), (8) integration with Istio service mesh for higher-level circuit breaking.

FAQ

Mini Project

Build a gRPC resilience framework: (1) client-side circuit breaker interceptor for unary calls with gRPC-specific status code mapping, (2) streaming interceptor that protects RecvMsg/SendMsg with circuit breaker state, (3) server-side circuit breaker that rejects requests after N concurrent requests or M failures, (4) fallback support: cached response proto for circuit open state, (5) Prometheus metrics exporter for all circuit breaker states and failure counts by status code, (6) health-checking service: the circuit breaker uses a dedicated health check RPC for half-open probes, (7) Istio DestinationRule with circuit breaker settings as the outer layer of protection.

What's Next

Continue with Message Queue to learn circuit breaker patterns for message brokers. Then explore Event-Driven Architecture for event-driven circuit breaker patterns.