OpenTelemetry Sampling Rate Not Working Fix
In this tutorial, you'll learn about OpenTelemetry Sampling Rate Not Working Fix. We cover key concepts, practical examples, and best practices.
Your OpenTelemetry instrumentation sends every single trace despite setting a sampling rate of 0.1 — the sampler configuration isn't applied, the wrong sampler type is used, or tail-based sampling overrides head-based decisions.
The Problem
# WRONG — setting sampler after the tracer provider is initialized
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.sampling import TraceIdRatioBased
provider = TracerProvider()
# Sampler not set — uses default (always sample)
trace.set_tracer_provider(provider)
# This has no effect — provider already created
provider.sampler = TraceIdRatioBased(0.1)
Every trace is sampled because the default sampler (ALWAYS_ON) was set when the provider was created. The sampler can only be set at initialization time.
Step-by-Step Fix
1. Set the sampler during provider creation
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.sampling import TraceIdRatioBased, ParentBased
# Correct: pass sampler to the constructor
provider = TracerProvider(
sampler=ParentBased(root=TraceIdRatioBased(0.1))
)
trace.set_tracer_provider(provider)
2. Use environment variables
# Sample 10% of traces
export OTEL_TRACES_SAMPLER=traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.1
# Always sample
export OTEL_TRACES_SAMPLER=always_on
# Never sample
export OTEL_TRACES_SAMPLER=always_off
3. Use ParentBased sampler for distributed traces
from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased
# If parent is sampled, sample this child. Otherwise, use ratio.
sampler = ParentBased(
root=TraceIdRatioBased(0.1) # Decision for root spans
)
This ensures that if a root span is sampled, all child spans (across services) are also sampled, giving you complete traces.
4. Configure tail-based sampling in the collector
# otel-collector-config.yaml
processors:
tail_sampling:
decision_wait: 30s
num_traces: 1000
policies:
# Sample all errors
- name: error-policy
type: status_code
status_code: { status_codes: [ERROR] }
# Sample slow traces
- name: slow-policy
type: latency
latency: { threshold_ms: 2000 }
# Sample 10% of remaining
- name: ratio-policy
type: probabilistic
probabilistic: { sampling_percentage: 10 }
5. Verify the sampling decision
# Check if a span is sampled
from opentelemetry import trace
with trace.get_tracer(__name__).start_as_current_span("test") as span:
if span.get_span_context().trace_flags.sampled:
print("This span is sampled")
Expected output:
Sampling rate: 0.1 (10%)
Total requests: 10000
Sampled traces: 987 (~9.87%)
✓ Sampling working as expected
Prevention Tips
- Set sampler via environment variables for consistent config across services
- Use
ParentBasedsampler to preserve trace completeness - Validate sampling with a small traffic test before production rollout
- Use tail-based sampling in the collector for error-focused sampling
- Monitor sampled/unsampled ratios in collector metrics
Common Mistakes with sampling
- Misunderstanding that
Stringis[Char]with poor performance for large text operations - Using
foldlinstead offoldl'causing stack overflow on large lists - Forgetting
deriving (Show, Eq)on custom data types needed for debugging
These mistakes appear frequently in real-world OPENTELEMETRY code. DodaTech's contributors have identified these patterns through analysis of open-source projects and production systems.
Practice Exercise
Write a pure function that safely divides two integers using Maybe, then test it with edge cases like division by zero and negative numbers.
This exercise reinforces the concepts covered in this guide. Try implementing it before checking online solutions.
FAQ
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro