gRPC Troubleshooting — Debugging Common Issues in gRPC Services

In this tutorial, you will learn about grpc troubleshooting. We cover key concepts, practical examples, and best practices to help you master this topic.

gRPC troubleshooting helps diagnose and resolve common issues like connection failures, TLS handshake errors, deadline exceeded errors, message size limits, streaming hangs, and client-server incompatibilities using specialized tools like grpcurl and gRPC debug utilities.

What You'll Learn

• Using grpcurl for ad-hoc gRPC debugging
• Diagnosing connection and TLS errors
• Fixing deadline exceeded and timeout issues
• Resolving message size limit errors
• Debugging streaming issues
• Protocol version and codec mismatches
• Enabling gRPC debug logging

Why It Matters

gRPC errors are often cryptic and hard to diagnose without the right tools. A connection reset or unexpected EOF can mean TLS mismatch, protocol version incompatibility, or a server crash. DodaTech's Durga Antivirus Pro reduces mean time to resolution (MTTR) from hours to minutes using standardized debugging procedures and tooling.

Real-World Use

A client's ReportThreat calls started failing with UNAVAILABLE. Using grpcurl to test the server revealed it was running but not responding on the expected port. The debug logs showed the server failed to bind because the port was already in use by a zombie Process.

flowchart TB
    A["Issue Detected"] --> B{"Error Type"}
    B -->|"UNAVAILABLE"| C["Check server running\nCheck port binding\nCheck firewall"]
    B -->|"UNAUTHENTICATED"| D["Check token expiry\nCheck signing key\nCheck cert chain"]
    B -->|"DEADLINE_EXCEEDED"| E["Increase timeout\nCheck network latency\nCheck server load"]
    B -->|"INTERNAL"| F["Check server logs\nCheck panic recovery\nCheck database"]
    B -->|"RESOURCE_EXHAUSTED"| G["Increase message size\nCheck rate limits\nCheck memory"]
    C --> H["Use grpcurl to test"]
    D --> H
    E --> H
    F --> H
    G --> H
    H --> I["Service Healthy"]

Code Examples

Example 1: Using grpcurl for Debugging

# Install grpcurl
go install github.com/fullstorydev/grpcurl/cmd/grpcurl@latest

# List all services on a server
grpcurl -plaintext localhost:50051 list

# List all methods on a service
grpcurl -plaintext localhost:50051 list threat.v1.ThreatService

# Invoke a method with JSON input
grpcurl -plaintext \
  -d '{"device_id": "dev-001", "threat_name": "Test"}' \
  localhost:50051 \
  threat.v1.ThreatService/ReportThreat

# With TLS
grpcurl -cacert ca.crt -cert client.crt -key client.key \
  -d '{}' \
  api.dodatech.com:443 \
  threat.v1.ThreatService/ListThreats

# Describe a message type
grpcurl -plaintext localhost:50051 describe .threat.v1.Threat

# Use reflection to discover services
grpcurl -plaintext localhost:50051 list

# Check server version
grpcurl -plaintext localhost:50051 \
  grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo

Example 2: Enabling gRPC Debug Logging

package main

import (
    "os"
    "google.golang.org/grpc/grpclog"
)

func enableDebugLogging() {
    // Set gRPC log level
    os.Setenv("GRPC_GO_LOG_SEVERITY_LEVEL", "info")
    os.Setenv("GRPC_GO_LOG_VERBOSITY_LEVEL", "2")
    
    // Use standard logger
    grpclog.SetLoggerV2(grpclog.NewLoggerV2WithVerbosity(
        os.Stdout, os.Stdout, os.Stderr, 99,
    ))
}

// For specific debugging, enable HTTP/2 frame logging
// export GRPC_TRACE=all
// export GRPC_VERBOSITY=DEBUG

# Python debug logging
import grpc
import logging

# Enable gRPC debug logging
logging.basicConfig(level=logging.DEBUG)
grpc._simple_stubs._LOGGER.setLevel(logging.DEBUG)

# Or via environment
# export GRPC_TRACE=all
# export GRPC_VERBOSITY=DEBUG

# Debug connection state
channel = grpc.insecure_channel("localhost:50051")
print(f"Channel state: {channel.get_state()}")
channel.subscribe(
    lambda state: print(f"State changed: {state}"),
    try_to_connect=True,
)

// Node.js debug logging
const grpc = require('@grpc/grpc-js');

// Enable trace logging
grpc.setLogger(console);
grpc.setLogVerbosity(grpc.logVerbosity.DEBUG);

// Trace specific components
// GRPC_TRACE=all for everything
// GRPC_TRACE=http,deadline for specific components
// GRPC_VERBOSITY=DEBUG for detailed messages

Example 3: Common Error Diagnostics Script

#!/bin/bash

# Comprehensive gRPC diagnostic script

SERVICE=$1
ADDRESS=${2:-localhost:50051}

echo "=== gRPC Diagnostic: $SERVICE at $ADDRESS ==="
echo ""

# 1. Check if port is open
echo "--- Port Check ---"
if nc -zv ${ADDRESS/:/ } 2>/dev/null; then
    echo "PASS: Port is open"
else
    echo "FAIL: Cannot connect to $ADDRESS"
    exit 1
fi

# 2. Check TLS
echo ""
echo "--- TLS Check ---"
if openssl s_client -connect $ADDRESS \
    -servername $SERVICE 2>/dev/null <<< "Q"; then
    echo "PASS: TLS handshake successful"
else
    echo "INFO: Server may not use TLS (try -plaintext)"
fi

# 3. Check gRPC reflection
echo ""
echo "--- Service Discovery ---"
if grpcurl -plaintext $ADDRESS list 2>/dev/null; then
    echo "PASS: gRPC reflection works"
else
    echo "FAIL: Cannot list services"
    echo "  - Server may not have reflection enabled"
    echo "  - Try: protoc ... --grpc_opt enable_reflection"
fi

# 4. Check health
echo ""
echo "--- Health Check ---"
grpcurl -plaintext -d '{"service": "'$SERVICE'"}' \
    $ADDRESS grpc.health.v1.Health/Check 2>/dev/null

# 5. Check message size
echo ""
echo "--- Message Size Test ---"
LARGE_PAYLOAD=$(python -c "print('A'*1024*1024)")
grpcurl -plaintext \
    -d "{\"threat_name\": \"$LARGE_PAYLOAD\"}" \
    $ADDRESS threat.v1.ThreatService/ReportThreat 2>&1 | \
    grep -q "RESOURCE_EXHAUSTED" && \
    echo "WARN: Message size limit hit" || \
    echo "PASS: 1MB message OK"

Common Mistakes

1. Ignoring gRPC debug logs — gRPC's internal debug logs often contain the exact error reason. Enable them before digging deeper.
2. Assuming network is fine without checking — a firewall rule, load balancer config, or DNS resolution can block gRPC. Always verify basic connectivity first.
3. Forgetting to check both sides — a client-side error can be caused by server-side misconfiguration. Check logs on both ends.
4. Not testing with grpcurl — grpcurl bypasses your client code and tests the server directly, isolating the problem to client or server.
5. Confusing gRPC status codes — UNAVAILABLE doesn't mean the server is down. It means the connection failed. This could be a proxy, load balancer, or network issue.

Practice Questions

1. What does grpcurl do and when should you use it?
2. How do you enable gRPC debug logging in different languages?
3. What is the first thing to check when you get UNAVAILABLE?
4. How do you diagnose a TLS handshake failure?
5. What tools can inspect HTTP/2 frames for gRPC debugging?

Challenge: Create a gRPC troubleshooting runbook that covers: connection failures, TLS errors, timeout issues, message size problems, streaming hangs, protocol mismatches, and server crashes. Include specific commands and log patterns to identify each issue.

Mini Project

Build a gRPC diagnostic CLI tool that: checks server connectivity, tests TLS handshake, lists available services via Reflection, invokes health check, tests message size limits, enables debug logging, and generates a diagnostic report with recommendations for common issues.

FAQ

What's Next

Learn gRPC best practices