Circuit Breaker in Go — Implementing Resilience with gobreaker and Custom Patterns

In this tutorial, you will learn about Circuit Breaker in Go. We cover key concepts, practical examples, and best practices to help you master this topic.

Go circuit breaker implementation using gobreaker provides lightweight, goroutine-safe state management with customizable failure thresholds, timeouts, and half-open probes, integrated with HTTP middleware and gRPC interceptors for microservice resilience.

flowchart TD
    Go[Go Service] -->|HTTP/gRPC| CB[Gobreaker
Circuit Breaker]
    CB -->|Closed| Client[HTTP Client/gRPC]
    CB -->|Open| Block[Block Request]
    CB -->|State Change| MW[Metrics Middleware]
    Client -->|Success| OK[Return Response]
    Client -->|Fail| Fail[Increment Counter]

What You'll Learn

• Gobreaker library usage
• HTTP middleware integration
• gRPC interceptor patterns
• Custom state management
• Prometheus metrics export

Why It Matters

Go Microservices need thread-safe, lightweight circuit breakers that integrate with Go's concurrency model. Gobreaker provides goroutine-safe circuit breakers with customizable state transitions and ready-to-use HTTP/gRPC middleware patterns.

Real-World Use

DodaTech's Go API Gateway uses gobreaker per downstream service with Prometheus metrics. When the inventory service fails, the circuit breaker opens within 10 requests and returns cached responses from Redis. The gateway processes 10K req/s with sub-millisecond circuit breaker overhead.

Basic Gobreaker Usage

package main

import (
    "fmt"
    "time"
    "errors"
    "github.com/sony/gobreaker"
)

func main() {
    cb := gobreaker.NewCircuitBreaker(gobreaker.Settings{
        Name:        "payment-service",
        MaxRequests: 3,
        Interval:    60 * time.Second,
        Timeout:     30 * time.Second,
        ReadyToTrip: func(counts gobreaker.Counts) bool {
            failureRatio := float64(counts.TotalFailures) / float64(counts.Requests)
            return counts.Requests >= 5 && failureRatio >= 0.6
        },
        OnStateChange: func(name string, from, to gobreaker.State) {
            fmt.Printf("Circuit '%s' changed from %s to %s\n", name, from, to)
        },
    })

    for i := 0; i < 10; i++ {
        result, err := cb.Execute(func() (interface{}, error) {
            return callPaymentService()
        })
        if err != nil {
            fmt.Printf("Request %d: %v\n", i+1, err)
        } else {
            fmt.Printf("Request %d: %v\n", i+1, result)
        }
        time.Sleep(100 * time.Millisecond)
    }
}

func callPaymentService() (string, error) {
    if time.Now().Unix()%2 == 0 {
        return "", errors.New("service unavailable")
    }
    return "payment successful", nil
}

Expected output:

Circuit 'payment-service' changed from closed to open
Request 1: payment successful
Request 2: service unavailable
...
Request 6: circuit breaker is open

HTTP Middleware

package main

import (
    "fmt"
    "net/http"
    "time"
    "github.com/sony/gobreaker"
)

func CircuitBreakerMiddleware(cb *gobreaker.CircuitBreaker, next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        result, err := cb.Execute(func() (interface{}, error) {
            recorder := &responseRecorder{ResponseWriter: w, statusCode: http.StatusOK}
            next.ServeHTTP(recorder, r)
            if recorder.statusCode >= 500 {
                return nil, fmt.Errorf("status %d", recorder.statusCode)
            }
            return recorder, nil
        })
        if err != nil {
            http.Error(w, "Service unavailable", http.StatusServiceUnavailable)
        }
        _ = result
    })
}

type responseRecorder struct {
    http.ResponseWriter
    statusCode int
}

func (r *responseRecorder) WriteHeader(code int) {
    r.statusCode = code
    r.ResponseWriter.WriteHeader(code)
}

func main() {
    cb := gobreaker.NewCircuitBreaker(gobreaker.Settings{
        Name:        "inventory-service",
        MaxRequests: 5,
        Timeout:     30 * time.Second,
        ReadyToTrip: func(counts gobreaker.Counts) bool {
            return counts.TotalFailures >= 3
        },
    })

    handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprintln(w, "Inventory data")
    })

    http.Handle("/api/inventory", CircuitBreakerMiddleware(cb, handler))
    fmt.Println("Server starting on :8080")
    http.ListenAndServe(":8080", nil)
}

Expected HTTP responses:

GET /api/inventory -> 200 "Inventory data" (normal)
GET /api/inventory -> 503 "Service unavailable" (circuit open)

gRPC Interceptor

package main

import (
    "fmt"
    "golang.org/x/net/context"
    "google.golang.org/grpc"
    "google.golang.org/grpc/codes"
    "google.golang.org/grpc/status"
    "github.com/sony/gobreaker"
)

func CircuitBreakerUnaryClientInterceptor(cb *gobreaker.CircuitBreaker) grpc.UnaryClientInterceptor {
    return func(ctx context.Context, method string, req, reply interface{},
        cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {

        _, err := cb.Execute(func() (interface{}, error) {
            return nil, invoker(ctx, method, req, reply, cc, opts...)
        })
        if err == gobreaker.ErrOpenState {
            return status.Error(codes.Unavailable, "circuit breaker open")
        }
        if err == gobreaker.ErrTooManyRequests {
            return status.Error(codes.ResourceExhausted, "circuit breaker rate limited")
        }
        return err
    }
}

func main() {
    cb := gobreaker.NewCircuitBreaker(gobreaker.Settings{
        Name: "product-service",
    })

    interceptor := CircuitBreakerUnaryClientInterceptor(cb)
    conn, _ := grpc.Dial("localhost:50051",
        grpc.WithUnaryInterceptor(interceptor),
        grpc.WithInsecure())
    defer conn.Close()

    fmt.Println("gRPC client with circuit breaker interceptor configured")
}

Expected output:

gRPC client with circuit breaker interceptor configured
[when circuit open] rpc error: code = Unavailable desc = circuit breaker open

Common Mistakes

• Interval too short -- short interval Windows reset failure counts quickly, preventing the circuit from opening. Set Interval based on typical traffic volume: 60-300 seconds for most services.
• MaxRequests too high in half-open -- setting MaxRequests to 100 in half-open floods the recovering service with probes. Set MaxRequests to 1-5 for conservative recovery testing.
• Not using ReadyToTrip for nuanced decisions -- default ReadyToTrip opens when consecutive failures exceed MaxRequests. Customize ReadyToTrip for failure ratio-based decisions, which is more stable than absolute counts.
• Ignoring goroutine safety -- circuit breaker state must be thread-safe. Gobreaker handles this internally. Custom implementations must use sync.RWMutex or atomic operations.
• No timeout on circuit breaker execute -- cb.Execute blocks if the wrapped function blocks. Always set context timeouts on the wrapped call. The circuit breaker does not impose its own execution timeout.

Practice Questions

1. How does gobreaker's Interval setting affect failure counting?
2. What is the purpose of MaxRequests in half-open state?
3. How do you implement a custom ReadyToTrip function?
4. What circuit breaker state changes should you log?
5. How do you integrate circuit breakers with gRPC interceptors?

Challenge

Build a Go circuit breaker framework: (1) HTTP middleware that wraps gobreaker and returns cached responses when open, (2) gRPC interceptor for both client and server with error code mapping, (3) Redis-backed distributed state for circuit breakers across multiple instances, (4) Prometheus metrics: circuit state, request count, failure count, state transitions, (5) dynamic configuration via HTTP endpoint (update thresholds without restart), (6) active health checking that probes the downstream service in half-open state with a separate timeout.

FAQ

Mini Project

Build a Go resilience middleware library: (1) configurable circuit breaker with gobreaker, (2) retry with exponential backoff (3 attempts, 100ms base), (3) timeout per downstream service, (4) HTTP client middleware that combines all three, (5) gRPC client interceptor with same capabilities, (6) Prometheus metrics for all operations, (7) health check endpoint showing circuit breaker states per service, (8) dynamic configuration via environment variables or config file.

What's Next

Continue with Async Patterns to learn async-compatible circuit breakers. Then explore Reactive Streams for reactive circuit breaker patterns.