Prometheus Service Discovery: Kubernetes, Consul & EC2

In this tutorial, you'll learn about Prometheus Service Discovery: Kubernetes, Consul & EC2. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Prometheus service discovery automatically detects targets to scrape based on integrations with Orchestration platforms, service registries, and cloud providers, eliminating the need to manually list every target in static configuration files.

What You'll Learn

In this tutorial, you will configure Prometheus service discovery for Kubernetes pods and services, Consul service registry, and AWS EC2 instances, and understand how relabeling rules customize the scraped targets.

Why It Matters

In dynamic environments, servers, containers, and services are constantly created and destroyed. Manually updating static_configs for every change is impossible. Service discovery ensures Prometheus always scrapes the right targets without human intervention, making monitoring as dynamic as the infrastructure it watches.

Real-World Use

Durga Antivirus Pro runs hundreds of scanner instances on AWS EC2, auto-scaled based on scan queue depth. Prometheus uses EC2 service discovery to automatically detect new scanner instances, scrape their metrics, and stop scraping terminated instances within minutes. No configuration changes are needed when the auto-scaling group adds or removes instances.

Kubernetes Service Discovery

Prometheus can discover Kubernetes pods, services, endpoints, and nodes. Use the Kubernetes_sd_configs block:

scrape_configs:
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        action: keep
        regex: my-app

The role field can be pod, service, endpoints, node, or ingress. Relabeling filters which discovered targets to keep.

Consul Service Discovery

For services registered with Consul, use consul_sd_configs:

scrape_configs:
  - job_name: consul-services
    consul_sd_configs:
      - server: localhost:8500
        services:
          - web
          - api
    relabel_configs:
      - source_labels: [__meta_consul_service]
        target_label: job

Prometheus queries the Consul API for registered services and their health status, automatically scraping only healthy service instances.

EC2 Service Discovery

For AWS EC2 instances, use ec2_sd_configs with AWS credentials:

scrape_configs:
  - job_name: ec2
    ec2_sd_configs:
      - region: us-east-1
        access_key: AKIA...
        secret_key: ...
        port: 9100
    relabel_configs:
      - source_labels: [__meta_ec2_tag_Name]
        target_label: instance_name

Prometheus queries the AWS EC2 API to discover running instances, using tags to filter and label targets. The port parameter sets the scrape port for all discovered instances.

Relabeling for Service Discovery

Relabeling is essential for service discovery. It lets you:

• Filter targets with keep or drop actions
• Map metadata labels to Prometheus labels
• Modify metric paths and scrape intervals per target

Common metadata labels include __meta_Kubernetes_pod_name, __meta_ec2_instance_id, and __meta_consul_service.

Common Mistakes

1. Missing IAM Permissions for Cloud Discovery

EC2 and other cloud SD configurations require specific IAM permissions. Without them, discovery returns no targets. Ensure the EC2:DescribeInstances permission is granted.

2. Not Using Relabeling to Filter Targets

Without relabeling, service discovery scrapes everything it finds. Use keep or drop to filter to only the targets you want.

3. Ignoring Target Labels from Metadata

Metadata labels contain valuable information (instance type, region, pod name). Map them to Prometheus labels for better querying.

4. No Health Check Filtering

For Consul, Prometheus discovers all services regardless of health status unless you configure allow_stale or filter by health. Always check service health.

5. Overloading the API

Setting a very short refresh_interval (below 30 seconds) for cloud or Consul SD causes excessive API calls and may result in Rate Limiting.

Practice Questions

1. What problem does service discovery solve for Prometheus? It automatically finds and scrapes targets in dynamic environments without manual configuration, adapting to infrastructure changes in real time.

2. What roles are available for Kubernetes service discovery? Pod, service, endpoints, node, and ingress. Each discovers different types of Kubernetes resources.

3. How does relabeling work with service discovery? Relabeling reads source labels (metadata from SD), applies actions (keep, drop, replace), and produces target labels for Prometheus.

4. What permissions are needed for EC2 service discovery? The ec2:DescribeInstances permission is required to list EC2 instances in the configured region.

5. Challenge: Set up Kubernetes service discovery that scrapes all pods with label app=api and adds the pod name as a Prometheus label, ignoring all other pods.

What's Next

Visualize your Prometheus metrics with Grafana dashboards for powerful, customizable monitoring views.

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.