Skip to content

OpenTelemetry Sampling Rate Not Working Fix

DodaTech Updated 2026-06-24 3 min read

In this tutorial, you'll learn about OpenTelemetry Sampling Rate Not Working Fix. We cover key concepts, practical examples, and best practices.

Your OpenTelemetry instrumentation sends every single trace despite setting a sampling rate of 0.1 — the sampler configuration isn't applied, the wrong sampler type is used, or tail-based sampling overrides head-based decisions.

The Problem

# WRONG — setting sampler after the tracer provider is initialized
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.sampling import TraceIdRatioBased

provider = TracerProvider()
# Sampler not set — uses default (always sample)
trace.set_tracer_provider(provider)

# This has no effect — provider already created
provider.sampler = TraceIdRatioBased(0.1)

Every trace is sampled because the default sampler (ALWAYS_ON) was set when the provider was created. The sampler can only be set at initialization time.

Step-by-Step Fix

1. Set the sampler during provider creation

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.sampling import TraceIdRatioBased, ParentBased

# Correct: pass sampler to the constructor
provider = TracerProvider(
    sampler=ParentBased(root=TraceIdRatioBased(0.1))
)
trace.set_tracer_provider(provider)

2. Use environment variables

# Sample 10% of traces
export OTEL_TRACES_SAMPLER=traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.1

# Always sample
export OTEL_TRACES_SAMPLER=always_on

# Never sample
export OTEL_TRACES_SAMPLER=always_off

3. Use ParentBased sampler for distributed traces

from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased

# If parent is sampled, sample this child. Otherwise, use ratio.
sampler = ParentBased(
    root=TraceIdRatioBased(0.1)  # Decision for root spans
)

This ensures that if a root span is sampled, all child spans (across services) are also sampled, giving you complete traces.

4. Configure tail-based sampling in the collector

# otel-collector-config.yaml
processors:
  tail_sampling:
    decision_wait: 30s
    num_traces: 1000
    policies:
      # Sample all errors
      - name: error-policy
        type: status_code
        status_code: { status_codes: [ERROR] }
      # Sample slow traces
      - name: slow-policy
        type: latency
        latency: { threshold_ms: 2000 }
      # Sample 10% of remaining
      - name: ratio-policy
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }

5. Verify the sampling decision

# Check if a span is sampled
from opentelemetry import trace

with trace.get_tracer(__name__).start_as_current_span("test") as span:
    if span.get_span_context().trace_flags.sampled:
        print("This span is sampled")

Expected output:

Sampling rate: 0.1 (10%)
Total requests: 10000
Sampled traces: 987 (~9.87%)
✓ Sampling working as expected

Prevention Tips

  • Set sampler via environment variables for consistent config across services
  • Use ParentBased sampler to preserve trace completeness
  • Validate sampling with a small traffic test before production rollout
  • Use tail-based sampling in the collector for error-focused sampling
  • Monitor sampled/unsampled ratios in collector metrics

Common Mistakes with sampling

  1. Misunderstanding that String is [Char] with poor performance for large text operations
  2. Using foldl instead of foldl' causing stack overflow on large lists
  3. Forgetting deriving (Show, Eq) on custom data types needed for debugging

These mistakes appear frequently in real-world OPENTELEMETRY code. DodaTech's contributors have identified these patterns through analysis of open-source projects and production systems.

Practice Exercise

Write a pure function that safely divides two integers using Maybe, then test it with edge cases like division by zero and negative numbers.

This exercise reinforces the concepts covered in this guide. Try implementing it before checking online solutions.

FAQ

### What's the difference between head-based and tail-based sampling?

Head-based sampling decides at the application level (first span) — simple but can't prioritize errors. Tail-based sampling decides in the collector after seeing the complete trace — can sample all errors and slow traces regardless of head decision. Use head-based for cost control, tail-based for error visibility.

Why does my sampling rate of 0.5 produce ~50% but not exactly 50%?

TraceIdRatioBased uses consistent hashing of the trace ID. For small numbers of traces, the actual ratio varies. For large numbers (10,000+), it converges to the configured rate. This is expected behavior — the sampler is probabilistic, not exact.

Can I use different sampling rates for different services?

Yes. Set OTEL_TRACES_SAMPLER_ARG differently per service. The ParentBased sampler ensures consistency: if a root span is sampled in Service A, child spans in Service B are sampled regardless of Service B's rate. Only root span decisions are rate-limited.

Built by the developers of DodaTech

Doda Browser, DodaZIP & Durga Antivirus Pro