Monitoring Tools: Prometheus, Grafana, Datadog & More

In this tutorial, you'll learn about Monitoring Tools: Prometheus, Grafana, Datadog & More. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Monitoring tools collect metrics, logs, and traces from your infrastructure to detect issues, track performance, and provide Observability into every layer of your stack.

What You'll Learn

• Core monitoring concepts: metrics, logs, and traces
• Setting up Prometheus for time-series metrics collection
• Building Grafana dashboards for visualization
• Configuring Datadog for SaaS-based monitoring
• Writing alerting rules and defining SLI/SLO targets

Why Monitoring Tools Matter

Without monitoring, you're flying blind. A server could be running at 100% CPU for hours before anyone notices — and by then, users have already left. Monitoring tools give you real-time visibility, historical trends, and automated alerts so you know about problems before your customers do. DodaTech uses Prometheus and Grafana to monitor Durga Antivirus Pro's update servers — tracking request rates, error rates, and resource usage across a global fleet of distribution nodes.

flowchart LR
    A[DevOps Basics] --> B[Monitoring Tools]
    B --> C[Metrics - Prometheus]
    B --> D[Logs - Elastic/Loki]
    B --> E[Traces - Jaeger]
    B --> F[Dashboards - Grafana]
    B --> G[Alerting]
    C --> H[Collect & Store]
    D --> I[Aggregate & Search]
    E --> J[Distributed Tracing]
    style B fill:#e6522c,color:#fff

Core Monitoring Concepts

Monitoring is built on three pillars: metrics (numbers over time), logs (event records), and traces (request paths through Distributed Systems).

Metrics with Prometheus

Prometheus scrapes metrics from targets at configured intervals and stores them in a time-series database.

# Install and run Prometheus with node_exporter
wget https://github.com/prometheus/prometheus/releases/download/v2.47.0/prometheus-2.47.0.linux-amd64.tar.gz
tar -xzf prometheus-*.tar.gz && cd prometheus-*

# Start Prometheus
./prometheus --config.file=prometheus.yml --storage.tsdb.retention.time=30d &

# Install node_exporter
wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz
tar -xzf node_exporter-*.tar.gz && cd node_exporter-*
./node_exporter &

# Verify metrics endpoint
curl http://localhost:9100/metrics | grep node_cpu_seconds
# Output:
# node_cpu_seconds_total{cpu="0",mode="idle"} 1234567.89
# node_cpu_seconds_total{cpu="0",mode="system"} 23456.78

PromQL Queries

PromQL is Prometheus's query language for slicing metric data:

# CPU usage percentage per instance
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory available percentage
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

# Request error rate (based on label status=5xx)
rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) * 100

# Output: time-series data points at each scrape interval
# {instance="web-1:9100"} 23.5

Grafana Dashboards

Grafana visualizes Prometheus metrics in real-time dashboards:

{
  "title": "Production Overview",
  "panels": [
    {
      "title": "CPU by Instance",
      "type": "graph",
      "targets": [{
        "expr": "100 - (avg by(instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100)",
        "legendFormat": "{{ instance }}"
      }]
    },
    {
      "title": "Request Rate",
      "type": "graph",
      "targets": [{
        "expr": "rate(http_requests_total[5m])",
        "legendFormat": "requests/s"
      }]
    }
  ]
}

Output: Grafana renders panels as time-series graphs. You can add alert thresholds, annotations, and template variables for dynamic dashboards.

Alerting Rules

groups:
  - name: production_alerts
    rules:
      - alert: HighCPU
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "CPU over 80% on {{ $labels.instance }}"

      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) * 100 > 5
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Error rate above 5% on {{ $labels.instance }}"

Datadog Integration

Datadog is a SaaS monitoring platform with built-in integrations for hundreds of services:

# Install Datadog agent on a server
DD_API_KEY=your_api_key DD_SITE="datadoghq.com" bash -c \
  "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script_agent7.sh)"

# Configure integrations in /etc/datadog-agent/conf.d/
# Example: postgres.yaml
init_config:
instances:
  - host: localhost
    port: 5432
    username: datadog
    password: your_password

Output: Datadog automatically discovers running services and begins collecting metrics. The web UI shows pre-built dashboards for PostgreSQL, NGINX, AWS, and 600+ integrations.

Choosing the Right Tool

Common Mistakes

1. Alert fatigue from poorly tuned thresholds: Setting alerts that fire too often trains teams to ignore them. Use for: clauses to require sustained violations, and set distinct severity levels.

2. Not using recording rules for expensive queries: Queries like histogram_quantile(0.99, ...) are slow. Pre-compute them with recording rules that run every scrape interval.

3. Scraping too frequently: A 1-second scrape interval on thousands of targets generates enormous data volume. 15-30 seconds is sufficient for most infrastructure metrics.

4. Ignoring label cardinality: Labels with many unique values (user IDs, email addresses) cause Prometheus to consume excessive memory. Keep cardinality under 100,000 per metric.

5. Not setting retention limits: Prometheus defaults to unlimited disk usage. Set --storage.tsdb.retention.time=30d and --storage.tsdb.retention.size=50GB.

Practice Questions

1. What are the three pillars of Observability? Answer: Metrics (numbers over time), logs (event records), and traces (request paths through Distributed Systems).

2. How does Prometheus collect metrics? Answer: Prometheus uses a pull model — it scrapes HTTP endpoints (targets) at configured intervals. Exporters translate third-party system metrics into Prometheus format.

3. What is the difference between a counter and a gauge in Prometheus? Answer: A counter only increases (request count, errors). A gauge goes up and down (CPU usage, memory). Use rate() on counters, not gauges.

4. When should you choose Datadog over Prometheus + Grafana? Answer: When you want an all-in-one SaaS solution with 600+ integrations, built-in APM, and less operational overhead managing your own monitoring stack.

Challenge

Set up a complete monitoring stack: install Prometheus and node_exporter on three servers, configure Grafana with a dashboard showing CPU/memory/disk across all servers, create alerting rules for high CPU (>80% for 10m) and low disk (<10%), route alerts to Slack via Alertmanager.

FAQ

Mini Project: Monitor a Web Application

# custom_exporter.py
# Expose custom application metrics for Prometheus
from prometheus_client import start_http_server, Counter, Histogram
import time
import random

REQUEST_COUNT = Counter('app_requests_total', 'Total requests', ['endpoint', 'status'])
REQUEST_DURATION = Histogram('app_request_duration_seconds', 'Request duration', ['endpoint'])

def process_request(endpoint):
    with REQUEST_DURATION.labels(endpoint=endpoint).time():
        time.sleep(random.uniform(0.01, 0.5))
        status = random.choice(['200', '200', '200', '500', '404'])
        REQUEST_COUNT.labels(endpoint=endpoint, status=status).inc()
        return status

if __name__ == '__main__':
    start_http_server(8000)
    while True:
        process_request('/api/users')
        process_request('/api/items')
        time.sleep(1)

Expected output: Prometheus scrapes http://localhost:8000/metrics for app_requests_total and app_request_duration_seconds. Grafana visualizes request rate (via rate(app_requests_total[5m])) and latency percentiles (via histogram_quantile(0.99, ...)).

What's Next

Related topics: Prometheus, Grafana, Datadog, SLI

What's Next

Congratulations on completing this Monitoring Tools tutorial! Here's where to go from here:

• Practice daily — Set up Prometheus monitoring on a personal project
• Build a project — Create a Grafana dashboard for your application
• Explore related topics — Check out centralized logging and SRE fundamentals

Remember: every expert was once a beginner. Keep coding!

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.