gRPC Health Check Protocol — Implementing and Consuming Health Probes

In this tutorial, you will learn about grpc health check protocol. We cover key concepts, practical examples, and best practices to help you master this topic.

The gRPC health check protocol defines a standard gRPC service for reporting server health, enabling load balancers, orchestrators, and clients to determine whether a service is ready to accept requests.

What You'll Learn

• The gRPC Health Checking Protocol (GRPC-HCP)
• Implementing a health server in Go and Python
• Kubernetes liveness and readiness probes for gRPC
• Client-side health checking and failover
• Custom health check logic (database, dependencies)

Why It Matters

Without health checks, a load balancer might route traffic to a dying server, causing errors for every request. The gRPC health check protocol provides a standardized way to report liveness (is the Process running?) and readiness (can it handle requests?). DodaTech's Durga Antivirus Pro uses health checks to monitor 50+ gRPC Microservices, automatically draining traffic from unhealthy instances within 5 seconds.

Real-World Use

A gRPC service for threat analysis depends on a database and a Machine Learning model. The health check pings both dependencies every 10 seconds. When the ML model fails to load, the health check returns NOT_SERVING, and Kubernetes stops routing traffic to that pod until it recovers.

flowchart TB
    A["K8s Prober"] --> B{Health Check}
    B --> C["Health Service\nCheck"]
    C --> D{"Dependencies\nHealthy?"}
    D -->|All OK| E["SERVING"]
    D -->|DB Down| F["NOT_SERVING"]
    D -->|Unknown| G["UNKNOWN"]
    E --> H["Route Traffic"]
    F --> I["Drain Connections"]
    style E fill:#bbf7d0,stroke:#16a34a
    style F fill:#fecaca,stroke:#dc2626

Code Examples

Example 1: Health Server in Go

package main

import (
    "context"
    "google.golang.org/grpc"
    "google.golang.org/grpc/health"
    "google.golang.org/grpc/health/grpc_health_v1"
)

func main() {
    s := grpc.NewServer()
    healthServer := health.NewServer()
    grpc_health_v1.RegisterHealthServer(s, healthServer)

    // Mark service as serving
    healthServer.SetServingStatus(
        "threat-service",
        grpc_health_v1.HealthCheckResponse_SERVING,
    )

    // Watch for shutdown
    go func() {
        <-ctx.Done()
        healthServer.SetServingStatus(
            "threat-service",
            grpc_health_v1.HealthCheckResponse_NOT_SERVING,
        )
        s.GracefulStop()
    }()
}

Example 2: Health Client in Python

import grpc
from grpc_health_v1 import health_pb2, health_pb2_grpc

def check_health(address, service="threat-service"):
    channel = grpc.insecure_channel(address)
    stub = health_pb2_grpc.HealthStub(channel)
    
    try:
        response = stub.Check(
            health_pb2.HealthCheckRequest(service=service),
            timeout=5,
        )
        status = response.status
        print(f"Health status for {service}: {status}")
        
        if status == health_pb2.HealthCheckResponse.SERVING:
            return True
        elif status == health_pb2.HealthCheckResponse.NOT_SERVING:
            print(f"Service {service} is not serving")
            return False
        else:
            print(f"Service {service} status unknown: {status}")
            return False
            
    except grpc.RpcError as e:
        print(f"Health check failed: {e.code()}")
        return False

# Watch health changes
def watch_health(address, service):
    channel = grpc.insecure_channel(address)
    stub = health_pb2_grpc.HealthStub(channel)
    request = health_pb2.HealthCheckRequest(service=service)
    
    for response in stub.Watch(request):
        print(f"Status changed: {response.status}")
        if response.status == health_pb2.HealthCheckResponse.SERVING:
            break

Example 3: Kubernetes Probe Configuration

apiVersion: v1
kind: Pod
metadata:
  name: grpc-threat-service
spec:
  containers:
  - name: threat-service
    image: dodatech/threat-service:latest
    ports:
    - containerPort: 50051
    livenessProbe:
      grpc:
        port: 50051
        service: threat-service
      initialDelaySeconds: 10
      periodSeconds: 15
    readinessProbe:
      grpc:
        port: 50051
        service: threat-service
      initialDelaySeconds: 5
      periodSeconds: 10
    startupProbe:
      grpc:
        port: 50051
        service: threat-service
      failureThreshold: 30
      periodSeconds: 10

Common Mistakes

1. Using TCP probes instead of gRPC health probes — TCP probes only check if the port is open, not if the service is actually ready. Use gRPC health probes for real readiness.
2. Not setting startup probes — without startup probes, Kubernetes starts liveness probes immediately, potentially killing slow-starting services before they're ready.
3. Marking as SERVING before dependencies are ready — the health check should report NOT_SERVING until all dependencies (DB, cache, ML model) are verified.
4. Not updating health status on shutdown — services should mark NOT_SERVING before graceful shutdown to prevent traffic loss during rolling updates.
5. Ignoring Watch-based health checks — the Watch RPC lets clients receive instant notifications instead of polling, reducing latency during failover.

Practice Questions

1. What is the difference between liveness and readiness probes?
2. How does the gRPC health check protocol differ from HTTP health checks?
3. Why should you use the Watch RPC instead of polling?
4. What status should a service return when its database is unreachable?
5. How do you implement custom health check logic for service-specific dependencies?

Challenge: Build a health check system for a multi-service gRPC architecture where each service reports its own health and also checks its downstream dependencies. Implement a health dashboard that aggregates all service statuses and alerts on failures.

Mini Project

Implement the gRPC health check protocol in a microservice with custom dependency checks (database, cache, downstream services). Configure Kubernetes probes (liveness, readiness, startup). Build a CLI health checker tool that watches multiple services and reports status changes.

FAQ

What's Next

Learn about gRPC metadata and headers