Prometheus Overview: Metrics, Monitoring & Alerting

Prometheus is an open-source monitoring and alerting toolkit originally built at SoundCloud that collects metrics from configured targets at given intervals, evaluates rule expressions, displays results, and triggers alerts when conditions become critical.

What You'll Learn

In this tutorial, you will learn how Prometheus works, its core architecture including the pull model and time-series database, the metrics it collects, and how it fits into a complete monitoring stack.

Why It Matters

Without monitoring, you are flying blind. Services fail, disks fill up, and latencies spike without anyone noticing. Prometheus gives you real-time visibility into every component of your system so you can detect and respond to problems before users are affected. Modern cloud-native infrastructure relies on Prometheus as the de facto standard for metrics collection.

Real-World Use

DigitalOcean uses Prometheus to monitor thousands of customer droplets and internal services. Durga Antivirus Pro uses Prometheus to track scan throughput, virus definition age, and API response times across its distributed scanning infrastructure, alerting the team when scan queues grow beyond safe thresholds.

Architecture Overview

Prometheus follows a pull-based architecture. The Prometheus server scrapes metrics from HTTP endpoints on monitored targets at regular intervals. Each target exposes metrics at an HTTP endpoint (usually /metrics). The server stores all scraped data in a local time-series database and evaluates recording and alerting rules against the data.

The main components are:

• Prometheus server -- scrapes and stores metrics
• Client libraries -- instrument application code
• Exporters -- expose metrics from third-party systems
• Alertmanager -- handles alerts and deduplication
• Grafana -- visualizes Prometheus data (optional but common)

Key Concepts

Metrics and Labels

A metric is a named measurement collected over time. Examples include http_requests_total, cpu_seconds_total, and memory_bytes. Each metric can have multiple labels (key-value pairs) that add dimensions:

http_requests_total{method="GET", endpoint="/api", status="200"} 1024

Labels allow you to slice and aggregate data (e.g., requests per endpoint or per status code).

Metric Types

Prometheus supports four core metric types:

Counter -- a cumulative value that only increases (total requests, total errors). Gauge -- a value that can go up or down (current memory usage, active connections). Histogram -- samples observations and counts them in configurable buckets (request latency, response sizes). Summary -- similar to histogram but calculates configurable quantiles over a sliding time window.

Pull Model vs Push Model

Prometheus pulls metrics from targets rather than waiting for targets to push. This design means:

• You can tell if a target is down (it stops responding to scrapes)
• Scrape intervals are controlled by the server, not the target
• Metrics collection does not depend on target availability

Common Mistakes

1. Using Too Many Labels

High-cardinality labels (like user IDs or email addresses) explode the number of time series and can crash Prometheus. Keep labels below 10 per metric.

2. Not Using the Pull Model

Some users try to push metrics to Prometheus using the Pushgateway for batch jobs. This is valid only for short-lived jobs. For long-running services, always use pull.

3. Ignoring Histogram Buckets

Default histogram buckets may not fit your use case. Always configure buckets that match your SLOs and expected latency ranges.

4. Forgetting Alertmanager

Alerting rules without Alertmanager will not send notifications. You must configure Alertmanager as the alert receiver.

5. No Retention Planning

By default, Prometheus retains data for 15 days. If you need longer retention, configure remote storage or use Thanos.

Practice Questions

1. What are the four core Prometheus metric types? Counter, Gauge, Histogram, and Summary. Counters only increase, gauges go up and down, histograms bucket observations, and summaries calculate quantiles.

2. Why does Prometheus use a pull model instead of a push model? The pull model lets the server detect down targets, control scrape intervals, and decouple collection from target availability.

3. What is the risk of high-cardinality labels? Too many unique label combinations create an explosion of time series, consuming excessive memory and potentially crashing the Prometheus server.

4. How do labels differ from metric names? A metric name identifies the measurement. Labels add dimensions for filtering and aggregation. The combination of metric name and labels defines a unique time series.

5. Challenge: Design a set of metrics and labels for a web application that tracks requests by endpoint, method, and status code, keeping cardinality under control.

What's Next

Set up Prometheus in your environment with the install guide, then learn PromQL to query your metrics effectively.

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.