OpenTelemetry — Complete Vendor-Neutral Instrumentation Guide

In this tutorial, you will learn about OpenTelemetry. We cover key concepts, practical examples, and best practices to help you master this topic.

OpenTelemetry (OTel) is a vendor-neutral standard for instrumenting applications to collect metrics, logs, and traces. It provides a single API that can export to any backend: Prometheus, Datadog, New Relic, Jaeger, or others.

What You'll Learn

You'll learn the OpenTelemetry architecture, how to instrument an API, and how to export data to multiple backends.

Why It Matters

Without OpenTelemetry, you are locked into a vendor's agent. OTel provides portable instrumentation that works with any Observability backend.

Real-World Use

A company uses OpenTelemetry to instrument all services. They export traces to Jaeger for debugging and metrics to Prometheus for alerting. When they migrate from Datadog to Grafana Cloud, they only change exporters.

Implementation

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.flask import FlaskInstrumentor

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(
    OTLPSpanExporter(endpoint="http://otel-collector:4317")
))
trace.set_tracer_provider(provider)

app = Flask(__name__)
FlaskInstrumentor().instrument_app(app)

@app.route("/api/users/<user_id>")
def get_user(user_id):
    with trace.get_tracer(__name__).start_as_current_span("get_user") as span:
        span.set_attribute("user_id", user_id)
        user = database.fetch_user(user_id)
        return jsonify(user)

# OpenTelemetry Collector configuration
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]

Common Mistakes

| Mistake | Fix | |---------|-----| | No sampling configuration | 100% tracing overhead in production | Configure probabilistic sampling (1-10%) | | Direct exporter instead of collector | Config changes require code changes | Always use OTel Collector | | Not using auto-instrumentation | Manual overhead for common frameworks | Use auto-instrumentation for Flask, Django, etc. | | Ignoring resource attributes | Cannot distinguish environments | Set service.name, deployment.environment | | No span error recording | Errors invisible in traces | Record exception events on spans |

Practice Questions

1. What problem does OpenTelemetry solve?
2. What is the OTel Collector?
3. What is auto-instrumentation?
4. How do you switch backends with OTel?
5. What is the difference between OTLP and direct exporters?

Challenge

Set up the OTel Collector with Docker. Instrument a Flask API with auto-instrumentation. Configure exporters for both Prometheus (metrics) and Jaeger (traces). View data in both backends.

What's Next

Learn about health check endpoints for API monitoring.