Blackbox Monitoring: External Probes with Prometheus

In this tutorial, you'll learn about Blackbox Monitoring: External Probes with Prometheus. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

What You Will Learn

This tutorial teaches you how to deploy the Prometheus Blackbox Exporter to monitor external endpoints, configure HTTP/HTTPS checks, TCP probes, ICMP pings, and DNS lookups, and alert on failures.

Why It Matters

Internal metrics tell you what your systems think about themselves. Blackbox monitoring tells you what your users actually experience. An API can return 200 OK internally but be unreachable from the outside due to a firewall misconfiguration, DNS issue, or CDN failure.

Real-World Use

The DodaTech team monitors the Doda Browser download mirrors from 10 locations worldwide using Blackbox Exporter. When one CDN endpoint started returning HTTP 503 errors, the probe detected it within 30 seconds and the team routed traffic to the healthy mirror -- before any user reported the issue.

The Blackbox Exporter is a Prometheus exporter that performs "blackbox" probes on external endpoints. It supports HTTP, HTTPS, TCP, ICMP, DNS, and gRPC checks. You configure the probe type in a module definition and Prometheus sends scrape requests with target parameters.

Prerequisites

• A running Prometheus instance (see Prometheus Introduction)
• Docker installed for the Blackbox Exporter
• A target endpoint to monitor (your own service or any public website)
• Basic Linux networking knowledge

Step-by-Step Tutorial

Step 1: Deploy the Blackbox Exporter

docker run -d --name blackbox_exporter \
  -p 9115:9115 \
  prom/blackbox-exporter:v0.25.0

Expected output: Blackbox Exporter listens on port 9115. Visit http://localhost:9115/probe?target=example.com&module=http_2xx to test.

Step 2: Configure Prometheus for Blackbox Monitoring

Add to <a href="/devops/prometheus-grafana/">Prometheus</a>.yml:

scrape_configs:
  - job_name: "blackbox-http"
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://example.com
          - https://api.dodatech.com
          - https://app.dodatech.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115

Step 3: Verify Probes in Prometheus

Restart Prometheus and check the Targets page. Look for the blackbox-http job. All targets should be UP.

probe_success

Expected output: A value of 1 for each target that responds successfully.

probe_duration_seconds

Expected output: The duration of each probe in seconds.

Step 4: Add TCP Probing

Add a new scrape config:

- job_name: "blackbox-tcp"
  metrics_path: /probe
  params:
    module: [tcp_connect]
  static_configs:
    - targets:
        - "api.dodatech.com:443"
        - "db.dodatech.com:5432"
        - "redis.dodatech.com:6379"
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: localhost:9115

Step 5: Add ICMP Probing (Ping)

- job_name: "blackbox-icmp"
  metrics_path: /probe
  params:
    module: [icmp]
  static_configs:
    - targets:
        - "8.8.8.8"
        - "1.1.1.1"
        - "gateway.dodatech.com"
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: localhost:9115

Step 6: Customize Probe Modules

Create a custom config.yml and mount it:

modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2"]
      valid_status_codes: [200, 201, 204]
      method: GET
      headers:
        User-Agent: "DodaTech-Probe/1.0"

  http_post_healthcheck:
    prober: http
    timeout: 10s
    http:
      method: POST
      headers:
        Content-Type: "application/json"
      body: '{"query": "health"}'
      valid_status_codes: [200]

  icmp:
    prober: icmp
    timeout: 5s
    icmp:
      preferred_ip_protocol: "ip4"

  tcp_connect:
    prober: tcp
    timeout: 5s

Run with custom config:

docker run -d --name blackbox_exporter \
  -p 9115:9115 \
  -v $(pwd)/config.yml:/config/blackbox.yml \
  prom/blackbox-exporter:v0.25.0 \
  --config.file=/config/blackbox.yml

Step 7: Probe with SSL Certificate Checking

modules:
  http_2xx_ssl:
    prober: http
    http:
      valid_status_codes: [200]
      fail_if_ssl: true
      tls_config:
        insecure_skip_verify: false

Query SSL certificate expiry:

probe_ssl_earliest_cert_expiry - time()

Expected output: Days remaining until the SSL certificate expires. Alert when this drops below 30.

Step 8: Alert on Probe Failures

groups:
  - name: blackbox
    rules:
      - alert: EndpointDown
        expr: probe_success == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Endpoint {{ $labels.instance }} is down"

      - alert: SSLCertExpiringSoon
        expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 30
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "SSL certificate for {{ $labels.instance }} expires in 30 days"

      - alert: HighLatency
        expr: probe_duration_seconds > 2
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High latency to {{ $labels.instance }}"

Learning Path

flowchart LR
    A[Blackbox Exporter] --> B[HTTP Probes]
    A --> C[TCP Probes]
    A --> D[ICMP Probes]
    A --> E[DNS Probes]
    B --> F[Prometheus]
    C --> F
    D --> F
    E --> F
    F --> G[Grafana Dashboard]
    F --> H[Alerts]
    style A fill:#4a90d9,color:#fff
    style H fill:#e67e22,color:#fff

Common Errors

1. Probe returns 0 for probe_success but endpoint is reachable -- The status code is outside the valid range. Check valid_status_codes in the module config.

2. ICMP probe fails with permission denied -- The Blackbox Exporter needs CAP_NET_RAW capability. Add --cap-add=NET_RAW to the Docker run command.

3. HTTP probe times out -- The timeout in the module config is too short. Increase to 10s or 30s for slow endpoints.

4. SSL certificate probe shows N/A -- The endpoint is not configured for HTTPS. Use the http_2xx module instead of an SSL-specific module.

5. Prometheus scrape returns 400 Bad Request -- The module parameter in the scrape config does not match a defined module name.

6. TCP probe connects but probe_success is 0 -- The module is looking for a specific response string. Use tcp_connect module for simple connectivity checks.

7. Too many probes cause Rate Limiting -- The target service has Rate Limiting. Reduce scrape frequency or increase scrape_interval.

Practice Questions

1. What is the difference between blackbox and whitebox monitoring? Answer: Blackbox monitors from the outside (what users see). Whitebox monitors from the inside (what the application reports about itself).

2. Which Blackbox Exporter module would you use to check if a port is open? Answer: The tcp_connect module, which attempts a TCP connection to the specified host and port.

3. How do you check SSL certificate expiry with Blackbox Exporter? Answer: Use the http_2xx module with SSL checking enabled, then query probe_ssl_earliest_cert_expiry.

4. Why does ICMP probing require special capabilities? Answer: ICMP ping requires raw socket access, which is restricted to privileged users. The CAP_NET_RAW capability grants this.

5. How do you pass the target URL to Blackbox Exporter? Answer: Through the __param_target relabel configuration, which maps the target address to the target query parameter.

Challenge

Set up Blackbox Exporter to monitor a production web application from three different perspectives. Configure HTTP probes that check for specific content in the response body, SSL expiry checking, and POST endpoint health checks. Add TCP probes for database and cache ports. Set up ICMP probes for network gateway and DNS servers. Create Prometheus alerting rules for: endpoint unreachable (critical, 1m), SSL cert expiring within 30 days (warning), latency above 2 seconds (warning), and packet loss above 10% (critical). Build a Grafana dashboard that shows probe status, latency trends, and SSL expiry calendar.

FAQ