gRPC Load Balancing — Client-Side, Proxy-Based, and Service Mesh Strategies
In this tutorial, you will learn about grpc Load Balancing. We cover key concepts, practical examples, and best practices to help you master this topic.
gRPC load balancing distributes RPC calls across multiple server instances using client-side algorithms (round-robin, weighted, least-loaded), proxy-based solutions (Envoy, NGINX), or service mesh sidecars (Istio, Linkerd).
What You'll Learn
- Client-side load balancing with round-robin
- Weighted load balancing for heterogeneous backends
- Proxy-based load balancing with Envoy
- Service mesh load balancing with Istio
- Health-aware load balancing
- Pick first vs round-robin vs custom policies
Why It Matters
gRPC's long-lived HTTP/2 connections break traditional TCP-level load balancing (kube-proxy). Without gRPC-aware load balancing, traffic concentrates on a few connections, leaving other servers idle. DodaTech's Durga Antivirus Pro uses client-side round-robin load balancing across 50+ gRPC Microservices, achieving 99% resource utilization across all instances.
Real-World Use
A gRPC threat analysis service runs 20 pods. With kube-proxy, all traffic goes to the first 5 pods (one TCP connection each). With client-side round-robin, traffic distributes evenly across all 20 pods. Each pod handles 5% of traffic instead of 20%.
flowchart TB
subgraph "Client-Side LB"
A["Client"] --> B["Round-Robin\nPicker"]
B --> C["Pod 1\n(25%)"]
B --> D["Pod 2\n(25%)"]
B --> E["Pod 3\n(25%)"]
B --> F["Pod 4\n(25%)"]
end
subgraph "Proxy-Based LB"
G["Client"] --> H["Envoy Proxy"]
H --> I["Pod 1"]
H --> J["Pod 2"]
H --> K["Pod 3"]
end
style B fill:#fef3c7,stroke:#d97706
style H fill:#fef3c7,stroke:#d97706
Code Examples
Example 1: Client-Side Round-Robin in Go
package main
import (
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
)
func createRoundRobinClient() *grpc.ClientConn {
// Service config enables round-robin
serviceConfig := `{
"loadBalancingConfig": [
{ "round_robin": {} }
],
"methodConfig": [{
"name": [{"service": "threat.v1.ThreatService"}],
"retryPolicy": {
"maxAttempts": 3,
"initialBackoff": "0.1s",
"maxBackoff": "1s",
"backoffMultiplier": 2,
"retryableStatusCodes": ["UNAVAILABLE"]
}
}]
}`
conn, _ := grpc.Dial(
"dns:///threat-service.default.svc.cluster.local:50051",
grpc.WithTransportCredentials(insecure.NewCredentials()),
grpc.WithDefaultServiceConfig(serviceConfig),
)
return conn
}
// Custom weighted round-robin picker
type WeightedPicker struct {
backends []*Backend
index int
mu sync.Mutex
}
type Backend struct {
Address string
Weight int
current int
}
func (p *WeightedPicker) Pick() string {
p.mu.Lock()
defer p.mu.Unlock()
total := 0
for _, b := range p.backends {
total += b.Weight
}
// Weighted round-robin
for {
backend := p.backends[p.index % len(p.backends)]
p.index++
backend.current++
if backend.current >= backend.Weight {
backend.current = 0
continue
}
return backend.Address
}
}
Example 2: Envoy Proxy Configuration
static_resources:
listeners:
- name: grpc_listener
address:
socket_address: { address: 0.0.0.0, port_value: 8080 }
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
codec_type: HTTP2
stat_prefix: grpc
route_config:
name: local_route
virtual_hosts:
- name: backend
domains: ["*"]
routes:
- match: { prefix: "/" }
route:
cluster: threat_service
# gRPC-specific timeout
max_stream_duration:
max_stream_duration: 30s
http_filters:
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
clusters:
- name: threat_service
type: STRICT_DNS
lb_policy: ROUND_ROBIN
typed_dns_resolver_config:
name: envoy.network.dns_resolver.cares
typed_extension_protocol_options:
envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
"@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
explicit_http_config:
http2_protocol_options: {}
load_assignment:
cluster_name: threat_service
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address: { address: 10.0.0.1, port_value: 50051 }
- endpoint:
address:
socket_address: { address: 10.0.0.2, port_value: 50051 }
Example 3: Istio Service Mesh Configuration
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: threat-service
spec:
host: threat-service.default.svc.cluster.local
trafficPolicy:
loadBalancer:
simple: ROUND_ROBIN
connectionPool:
http:
http2MaxRequests: 1000
maxRequestsPerConnection: 100
outlierDetection:
consecutive5xxErrors: 5
interval: 30s
baseEjectionTime: 30s
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: threat-service
spec:
hosts:
- threat-service
http:
- match:
- headers:
x-debug:
exact: "true"
route:
- destination:
host: threat-service
subset: v1
- route:
- destination:
host: threat-service
subset: v1
weight: 90
- destination:
host: threat-service
subset: v2
weight: 10
Common Mistakes
- Using kube-proxy for gRPC — kube-proxy load balances per TCP connection, not per RPC. With HTTP/2 multiplexing, one TCP connection handles many RPCs, so all traffic goes to one pod.
- Not using health-aware load balancing — a pod that's restarting but still in the DNS rotation will receive traffic and fail. Use readiness probes and endpoint watchers.
- Sticky load balancing without session affinity — if your service requires state, use consistent hashing instead of round-robin. Round-robin may route a user's requests to different backends.
- Setting timeouts too short in proxies — Envoy's stream timeout defaults to infinity. Set a reasonable timeout (30s) to prevent stuck connections from accumulating.
- Mixing multiple load balancing layers — client-side LB + kube-proxy + Envoy means 3 layers of load balancing, which can interact poorly. Choose one approach.
Practice Questions
- Why does kube-proxy not work for gRPC load balancing?
- How does client-side round-robin distribute traffic?
- What is the advantage of proxy-based load balancing with Envoy?
- How does Istio's virtual service support canary deployments?
- What is outlier detection and why is it important?
Challenge: Design a load balancing Strategy for a gRPC service with 50 instances across 3 regions. Include client-side round-robin for local traffic, failover to other regions, weighted distribution for canary deployments, and outlier detection for unhealthy instances.
Mini Project
Build a load balancing solution for gRPC with: client-side round-robin with DNS resolution, health-aware instance selection, weighted routing for canary deployments, circuit breaker for failing instances, and metrics for connection distribution and request success rates.
FAQ
What's Next
Learn more about gRPC load balancing
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro