Skip to content

gRPC Monitoring and Observability — Metrics, Dashboards, and Alerts

DodaTech Updated 2026-06-28 5 min read

In this tutorial, you will learn about grpc monitoring and observability. We cover key concepts, practical examples, and best practices to help you master this topic.

gRPC monitoring and observability uses structured metrics, logs, and traces to provide real-time visibility into service health, request patterns, error rates, and latency distributions.

What You'll Learn

  • Exposing gRPC metrics via Prometheus
  • Building Grafana dashboards for gRPC
  • Alerting on error rate and latency thresholds
  • Monitoring streaming connections
  • gRPC-specific RED metrics (Rate, Errors, Duration)
  • Observability with logs, metrics, and traces

Why It Matters

Without monitoring, you're flying blind. A gradual increase in error rates or latency goes unnoticed until users complain. DodaTech's Durga Antivirus Pro monitors every gRPC service with Prometheus metrics and Grafana dashboards, alerting the on-call team within 60 seconds of any anomaly.

Real-World Use

The monitoring dashboard shows that ReportThreat errors jumped from 0.1% to 5%. The alert triggers a PagerDuty notification. The on-call engineer sees the error is a database connection timeout. They restart the database Connection Pool, and errors return to normal within 2 minutes.

flowchart TB
    A["gRPC Service"] --> B["Metrics\nInterceptor"]
    B --> C["Prometheus\n/metrics endpoint"]
    C --> D["Grafana\nDashboard"]
    C --> E["Alertmanager"]
    E --> F{"Error Rate > 1%?"}
    E --> G{"p99 > 500ms?"}
    F -->|Yes| H["PagerDuty"]
    G -->|Yes| H
    H --> I["On-Call Engineer"]
    style H fill:#fecaca,stroke:#dc2626

Code Examples

Example 1: Prometheus Metrics Interceptor in Go

package main

import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
    "google.golang.org/grpc"
    "net/http"
)

var (
    grpcRequestsTotal = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "grpc_requests_total",
            Help: "Total gRPC requests",
        },
        []string{"method", "status"},
    )
    
    grpcRequestDuration = prometheus.NewHistogramVec(
        prometheus.HistogramOpts{
            Name: "grpc_request_duration_seconds",
            Help: "gRPC request duration",
            Buckets: []float64{
                0.001, 0.005, 0.01, 0.05,
                0.1, 0.5, 1, 2, 5,
            },
        },
        []string{"method"},
    )
    
    grpcRequestsInFlight = prometheus.NewGaugeVec(
        prometheus.GaugeOpts{
            Name: "grpc_requests_in_flight",
            Help: "Current in-flight gRPC requests",
        },
        []string{"method"},
    )
    
    grpcStreamMessagesTotal = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "grpc_stream_messages_total",
            Help: "Total stream messages sent/received",
        },
        []string{"method", "direction"},
    )
)

func init() {
    prometheus.MustRegister(grpcRequestsTotal)
    prometheus.MustRegister(grpcRequestDuration)
    prometheus.MustRegister(grpcRequestsInFlight)
    prometheus.MustRegister(grpcStreamMessagesTotal)
}

func metricsInterceptor(ctx context.Context,
    req interface{},
    info *grpc.UnaryServerInfo,
    handler grpc.UnaryHandler,
) (interface{}, error) {
    
    grpcRequestsInFlight.WithLabelValues(
        info.FullMethod).Inc()
    defer grpcRequestsInFlight.WithLabelValues(
        info.FullMethod).Dec()
    
    timer := prometheus.NewTimer(
        grpcRequestDuration.WithLabelValues(
            info.FullMethod))
    defer timer.ObserveDuration()
    
    resp, err := handler(ctx, req)
    
    status := "success"
    if err != nil {
        status = "error"
    }
    grpcRequestsTotal.WithLabelValues(
        info.FullMethod, status).Inc()
    
    return resp, err
}

// Metrics HTTP server
func metricsServer() {
    http.Handle("/metrics", promhttp.Handler())
    http.ListenAndServe(":9090", nil)
}

Example 2: Python Metrics Exporter

from prometheus_client import Counter, Histogram, Gauge, generate_latest
from grpc_interceptor import ServerInterceptor
import time

# Metrics definitions
grpc_requests_total = Counter(
    'grpc_requests_total',
    'Total gRPC requests',
    ['method', 'status'],
)

grpc_request_duration = Histogram(
    'grpc_request_duration_seconds',
    'gRPC request duration in seconds',
    ['method'],
    buckets=(0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1, 2, 5),
)

grpc_requests_in_flight = Gauge(
    'grpc_requests_in_flight',
    'Current in-flight gRPC requests',
    ['method'],
)

class MetricsInterceptor(ServerInterceptor):
    def intercept(self, method, request, context, method_name):
        grpc_requests_in_flight.labels(method=method_name).inc()
        start = time.time()
        
        try:
            response = method(request, context)
            grpc_requests_total.labels(
                method=method_name, status="success").inc()
            return response
        except Exception as e:
            grpc_requests_total.labels(
                method=method_name, status="error").inc()
            raise
        finally:
            grpc_requests_in_flight.labels(
                method=method_name).dec()
            grpc_request_duration.labels(
                method=method_name).observe(
                    time.time() - start)

# Health endpoint with metrics
from flask import Flask, Response
app = Flask(__name__)

@app.route("/metrics")
def metrics():
    return Response(
        generate_latest(),
        mimetype="text/plain",
    )

@app.route("/health")
def health():
    return {"status": "healthy"}

Example 3: Grafana Dashboard Configuration

{
  "title": "gRPC Service Dashboard",
  "panels": [
    {
      "title": "Request Rate (RPS)",
      "type": "graph",
      "targets": [{
        "expr": "sum(rate(grpc_requests_total[1m])) by (method)",
        "legendFormat": "{{ method }}"
      }]
    },
    {
      "title": "Error Rate (%)",
      "type": "graph",
      "targets": [{
        "expr": "sum(rate(grpc_requests_total{status=\"error\"}[1m])) / sum(rate(grpc_requests_total[1m])) * 100",
        "legendFormat": "Error %"
      }]
    },
    {
      "title": "P50/P95/P99 Latency",
      "type": "graph",
      "targets": [
        {
          "expr": "histogram_quantile(0.50, sum(rate(grpc_request_duration_seconds_bucket[5m])) by (le))",
          "legendFormat": "P50"
        },
        {
          "expr": "histogram_quantile(0.95, sum(rate(grpc_request_duration_seconds_bucket[5m])) by (le))",
          "legendFormat": "P95"
        },
        {
          "expr": "histogram_quantile(0.99, sum(rate(grpc_request_duration_seconds_bucket[5m])) by (le))",
          "legendFormat": "P99"
        }
      ]
    },
    {
      "title": "In-Flight Requests",
      "type": "graph",
      "targets": [{
        "expr": "sum(grpc_requests_in_flight) by (method)",
        "legendFormat": "{{ method }}"
      }]
    },
    {
      "title": "Stream Messages",
      "type": "graph",
      "targets": [{
        "expr": "sum(rate(grpc_stream_messages_total[1m])) by (direction)",
        "legendFormat": "{{ direction }}"
      }]
    }
  ]
}

Common Mistakes

  1. Not exposing metrics at all — gRPC services need explicit metrics instrumentation. Prometheus doesn't automatically capture gRPC metrics.
  2. Using too many label values — high-cardinality labels (like user ID) create millions of time series. Use low-cardinality labels: method, status, service name.
  3. Not tracking streaming metrics — streaming RPCs have different behavior than unary. Track message rate, stream duration, and active streams separately.
  4. Setting alert thresholds too tight — alert fatigue causes ignored alerts. Set thresholds at p99 + 3x standard deviation, not at the first sign of deviation.
  5. Not monitoring client-side metrics — server metrics show one side of the picture. Monitor client-side request rates and errors to catch network issues.

Practice Questions

  1. What are the RED metrics for gRPC monitoring?
  2. How do you expose Prometheus metrics from a gRPC service?
  3. What alert thresholds are appropriate for gRPC error rates?
  4. How do you monitor streaming RPCs differently from unary?
  5. Why is high label cardinality a problem in Prometheus?

Challenge: Design a monitoring and alerting Strategy for a gRPC microservice with: 5 services, 20 methods each, mix of unary and streaming RPCs, target p99 < 200ms, error rate < 0.5%, and automatic scaling trigger at 70% CPU.

Mini Project

Build a complete observability stack for a gRPC service with: Prometheus metrics exporter interceptor, Grafana dashboard with RED metrics, alerting rules for error rate and latency, streaming-specific metrics (active streams, message rate), and health check endpoint for Kubernetes.

FAQ

What is the minimum set of gRPC metrics I need?

The RED metrics: Rate (requests/sec), Errors (error count/rate), Duration (latency histogram). Add in-flight requests and stream counts for completeness.

How do I set up Prometheus for gRPC?

Add a metrics interceptor that records request count, duration, and errors. Expose a /metrics HTTP endpoint. Configure Prometheus to scrape it.

What alert thresholds should I use?

Warning: p95 > 200ms or error rate > 1%. Critical: p99 > 500ms or error rate > 5%. Adjust based on your service's baseline.

How do I monitor gRPC streaming performance?

Track: active stream count, messages per second per stream, stream duration, messages per stream, and stream error rate. Streaming has different performance characteristics.

Can I monitor gRPC with no code changes?

Some tools (like Istio/Envoy) can capture gRPC metrics at the proxy level, but service-level metrics (method, status) require interceptor instrumentation.

What's Next

Learn about gRPC logging

← Previous gRPC Tracing — Distributed Tracing with OpenTelemetry for gRPC Services Next → gRPC Logging — Structured Logging Patterns for gRPC Services

Built by the developers of DodaTech

Doda Browser, DodaZIP & Durga Antivirus Pro

Home Browse Grpc