Custom Metrics with Prometheus Exporters: Build Your Own

In this tutorial, you'll learn about Custom Metrics with Prometheus Exporters: Build Your Own. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

What You Will Learn

This tutorial teaches you how to build custom Prometheus exporters in Python and Go, implement metric types (counter, gauge, histogram), handle exporter best practices, and test your exporter end-to-end.

Why It Matters

Pre-built exporters cover most common systems, but every application has unique business metrics that are invisible to generic exporters -- items in a queue, active user sessions, API rate limits, or custom business KPIs. A custom exporter makes those metrics visible.

Real-World Use

The Durga Antivirus Pro team built a custom exporter that exposes the number of virus signatures in the database, the age of the last signature update, and the scan throughput per engine instance. These metrics helped them detect a stalled signature update pipeline before any antivirus definitions went stale.

A Prometheus exporter is an HTTP server that exposes metrics at the /metrics endpoint in plain-text format. The Prometheus client libraries handle the protocol details -- you just define and update your metrics. The Python <a href="/devops/prometheus-grafana/">Prometheus</a>_client library and the Go <a href="/devops/prometheus-grafana/">Prometheus</a>/client_golang are the most popular choices.

Prerequisites

• A running Prometheus instance (see Prometheus Introduction)
• Python 3.8+ with virtualenv
• Go 1.21+ (optional, for the Go example)
• Basic programming knowledge

Step-by-Step Tutorial

Step 1: Python Exporter -- Basic Setup

mkdir custom-exporter && cd custom-exporter
python3 -m venv venv
source venv/bin/activate
pip install prometheus-client flask

Step 2: Build a Weather Metrics Exporter

Create weather_exporter.py:

from prometheus_client import start_http_server, Gauge, Counter, Histogram
import time
import random

TEMPERATURE = Gauge("weather_temperature_celsius", "Current temperature in Celsius", ["city"])
HUMIDITY = Gauge("weather_humidity_percent", "Current humidity percentage", ["city"])
API_CALLS = Counter("weather_api_calls_total", "Total API calls", ["city", "status"])
REQUEST_DURATION = Histogram("weather_request_duration_seconds", "Duration of weather API requests")

cities = ["mumbai", "london", "tokyo", "new_york", "berlin"]

def fetch_weather(city):
    with REQUEST_DURATION.time():
        time.sleep(random.uniform(0.05, 0.2))
        temp = random.uniform(-5, 40)
        humidity = random.uniform(20, 95)
        success = random.random() > 0.1
        return temp, humidity, success

if __name__ == "__main__":
    start_http_server(8000)
    while True:
        for city in cities:
            try:
                temp, humidity, success = fetch_weather(city)
                TEMPERATURE.labels(city=city).set(temp)
                HUMIDITY.labels(city=city).set(humidity)
                status = "success" if success else "failed"
                API_CALLS.labels(city=city, status=status).inc()
            except Exception as e:
                API_CALLS.labels(city=city, status="exception").inc()
            time.sleep(1)
        time.sleep(10)

Expected output: Run the exporter, visit http://localhost:8000/metrics, and see your custom metrics.

python weather_exporter.py

Step 3: Query Your Custom Metrics

curl http://localhost:8000/metrics | grep weather

Output:

# HELP weather_temperature_celsius Current temperature in Celsius
# TYPE weather_temperature_celsius gauge
weather_temperature_celsius{city="mumbai"} 32.5
weather_temperature_celsius{city="london"} 12.3
# HELP weather_api_calls_total Total API calls
# TYPE weather_api_calls_total counter
weather_api_calls_total{city="mumbai",status="success"} 15.0
weather_api_calls_total{city="mumbai",status="failed"} 2.0

Step 4: Build a Go Exporter

mkdir go-exporter && cd go-exporter
go mod init go-exporter

Create main.go:

package main

import (
    "net/http"
    "time"
    "math/rand"

    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
    queueLength = prometheus.NewGaugeVec(
        prometheus.GaugeOpts{
            Name: "queue_items_total",
            Help: "Current number of items in the queue",
        },
        []string{"queue_name"},
    )
    jobsProcessed = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "jobs_processed_total",
            Help: "Total number of jobs processed",
        },
        []string{"queue_name", "status"},
    )
)

func collectMetrics() {
    for {
        queueLength.WithLabelValues("main_queue").Set(float64(rand.Intn(1000)))
        queueLength.WithLabelValues("retry_queue").Set(float64(rand.Intn(100)))
        jobsProcessed.WithLabelValues("main_queue", "success").Add(float64(rand.Intn(10)))
        jobsProcessed.WithLabelValues("main_queue", "failed").Add(float64(rand.Intn(3)))
        time.Sleep(15 * time.Second)
    }
}

func main() {
    prometheus.MustRegister(queueLength, jobsProcessed)
    go collectMetrics()
    http.Handle("/metrics", promhttp.Handler())
    http.ListenAndServe(":8001", nil)
}

go run main.go

Step 5: Add Prometheus Scrape Configuration

scrape_configs:
  - job_name: "custom-exporters"
    static_configs:
      - targets:
          - "localhost:8000"
          - "localhost:8001"

Step 6: Add Exporter Health Check

from prometheus_client import start_http_server, Gauge, Info
import psutil

EXPORTER_UP = Gauge("exporter_up", "Is the exporter itself healthy", ["exporter_name"])
MEMORY_USAGE = Gauge("exporter_process_memory_bytes", "Memory used by the exporter process", ["exporter_name"])

if __name__ == "__main__":
    start_http_server(8002)
    while True:
        EXPORTER_UP.labels(exporter_name="weather").set(1)
        MEMORY_USAGE.labels(exporter_name="weather").set(
            psutil.Process().memory_info().rss
        )
        time.sleep(30)

Step 7: Handle Errors Gracefully

import requests
from prometheus_client import Gauge

def fetch_metric_safe(metric, labels, fetcher, fallback=0):
    try:
        value = fetcher()
        metric.labels(**labels).set(value)
    except requests.exceptions.Timeout:
        metric.labels(**labels).set(fallback)
    except Exception as e:
        metric.labels(**labels).set(-1)

Step 8: Test the Exporter Endpoint

import requests

def test_exporter(url="http://localhost:8000/metrics"):
    response = requests.get(url)
    assert response.status_code == 200
    assert "weather_temperature_celsius" in response.text
    print("Exporter is healthy and returning expected metrics")

test_exporter()

Learning Path

flowchart LR
    A[Identify Business Metrics] --> B[Choose Language]
    B --> C[Python Exporter]
    B --> D[Go Exporter]
    C --> E[Metric Types]
    D --> E
    E --> F[/metrics Endpoint]
    F --> G[Prometheus Scrape]
    G --> H[Dashboards & Alerts]
    style A fill:#4a90d9,color:#fff
    style F fill:#e67e22,color:#fff

Common Errors

1. Prometheus cannot scrape the exporter -- The exporter is listening on the wrong interface. Bind to 0.0.0.0 instead of 127.0.0.1.

2. Metrics page returns 404 -- The /metrics endpoint is not registered. Verify the HTTP route is set to /metrics.

3. Counter values reset after restart -- Counters are in-memory and reset on restart. Use a persistent backend or accept that resets are normal for counters.

4. High cardinality from user-specific labels -- Labels like user_id or session_id create unbounded cardinality. Use total metrics and break down by broader categories.

5. Gauge value never updates -- The collection loop is not running or is sleeping too long. Verify the background loop logic.

6. Histogram buckets are not useful -- The default buckets do not match your latency distribution. Configure custom buckets: Histogram("latency", "Description", buckets=[0.1, 0.5, 1, 2, 5]).

7. Port already in use error -- Another process is using the same port. Change the port number or stop the other process.

Practice Questions

1. What format does a Prometheus exporter endpoint return? Answer: Plain-text with # HELP, # TYPE, and metric lines following the Prometheus exposition format.

2. When should you use a Gauge instead of a Counter? Answer: Use a Gauge for values that go up and down (temperature, memory). Use a Counter for values that only increase (request count, errors).

3. How do you add labels to a Prometheus metric? Answer: By specifying labels=["label1", "label2"] in the metric constructor and using .labels(label1="value1") when setting values.

4. What is the purpose of the start_http_server function in the Python client? Answer: It starts a background HTTP server in a separate thread that serves the /metrics endpoint.

5. Why should you avoid high-cardinality labels in exporters? Answer: Each unique label combination creates a new time series. High cardinality causes excessive memory and storage usage in Prometheus.

Challenge

Build a custom exporter that monitors an external API of your choice (weather API, GitHub API, cryptocurrency price API, or your own application). The exporter must expose at least three metric types: a gauge for the current value, a counter for total requests, and a histogram for request latency. Add labels for the endpoint and status code. Include a health check endpoint at /health. Write unit tests for the metric collection functions. Configure Prometheus to scrape your exporter. Create a Grafana dashboard showing: current value, request rate, error rate, and latency p50/p95/p99. Deploy the exporter as a Docker container.

FAQ