Prometheus Production: Retention, High Availability & Thanos

In this tutorial, you'll learn about Prometheus Production: Retention, High Availability & Thanos. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Running Prometheus in production requires configuring retention policies for storage management, setting up high availability to prevent data loss, and deploying Thanos to extend storage retention and provide a global query view across multiple Prometheus servers.

What You'll Learn

In this tutorial, you will configure Prometheus retention and storage for production, set up high availability with two Prometheus instances, deploy Thanos for long-term storage, and query across clusters.

Why It Matters

A single Prometheus server has limitations: 15-day default retention, no built-in high availability, and no horizontal scaling. In production, you need months of data for trend analysis, failover when a Prometheus server goes down, and the ability to query across multiple data centers. Thanos solves these challenges while keeping Prometheus as your core metrics collector.

Real-World Use

Durga Antivirus Pro operates Prometheus in three data centers with Thanos. Each data center runs two Prometheus replicas for HA. Thanos Sidecar uploads blocks to S3 for long-term storage. The operations team queries all data centers from a single Grafana dashboard using Thanos Query, and can investigate incidents from 6 months ago using data stored in S3.

Step 1: Configure Retention and Storage

Set retention by time and by data size in <a href="/devops/prometheus-grafana/">Prometheus</a>.yml:

storage:
  tsdb:
    retention:
      time: 30d
      size: 50GB

Or use command-line flags:

prometheus --storage.tsdb.retention.time=30d --storage.tsdb.retention.size=50GB

Use SSDs for faster write performance and configure remote storage for backup if long-term retention is needed.

Step 2: Set Up High Availability

Run two identical Prometheus instances scraping the same targets with the same configuration:

# Instance A -- port 9090
# Instance B -- port 9091
# Both use identical prometheus.yml

In Grafana, configure both as data sources. Use a load balancer to distribute queries or use Thanos Query to deduplicate the data. For Alertmanager, configure hashmod to prevent duplicate alerts:

alertmanager:
  cluster:
    enabled: true
  config:
    route:
      receiver: default

Step 3: Deploy Thanos Sidecar

Thanos Sidecar runs alongside each Prometheus instance, uploading TSDB blocks to object storage:

thanos sidecar \
  --tsdb.path /var/lib/prometheus \
  --objstore.config-file bucket.yaml \
  --prometheus.url http://localhost:9090

bucket.yaml configures the object store:

type: S3
config:
  bucket: prometheus-metrics
  endpoint: s3.amazonaws.com
  region: us-east-1

Step 4: Deploy Thanos Query

Thanos Query provides a global view across all Prometheus instances and stores:

thanos query \
  --store 10.0.1.10:10901 \
  --store 10.0.2.10:10901 \
  --store thanos-store:10901

Point Grafana to the Thanos Query endpoint (default port 9090) instead of individual Prometheus servers.

Step 5: Deploy Thanos Compactor

Thanos Compactor deduplicates, downsamples, and compacts TSDB blocks in object storage:

thanos compact \
  --objstore.config-file bucket.yaml \
  --data-dir /var/thanos/compact \
  --retention.resolution-raw=30d \
  --retention.resolution-5m=180d \
  --retention.resolution-1h=3y

This keeps raw data for 30 days, 5-minute downsampled data for 180 days, and hourly data for 3 years.

Common Mistakes

1. Not Testing Retention Deletion

Configured retention silently deletes data. Ensure your retention periods match Compliance requirements before data is permanently removed.

2. Running Prometheus Without HA

A single Prometheus instance is a single point of failure. Always run two replicas in production, even if you only use one for querying.

3. Object Store Without Versioning

Corruption in the object store can destroy all historical data. Enable bucket versioning or use a backup policy for your Thanos bucket.

4. Ignoring Thanos Store Memory

Thanos Store Gateway caches data in memory. Insufficient memory causes slow queries for historical data. Allocate at least 4GB per store instance.

5. Not Monitoring Prometheus Itself

Prometheus exposes its own metrics at /metrics. Monitor them with another Prometheus or a separate monitoring system to avoid circular dependencies.

Practice Questions

1. How do you configure retention by time and size in Prometheus? Use --storage.tsdb.retention.time and --storage.tsdb.retention.size flags or the equivalent config file settings.

2. What is the simplest approach to Prometheus high availability? Run two identical Prometheus instances scraping the same targets. Use Thanos Query or Grafana to handle deduplication.

3. What does Thanos Sidecar do? It runs alongside Prometheus and uploads completed TSDB blocks to object storage for long-term retention.

4. How does Thanos Compactor manage data retention? It deduplicates and downsamples data, then applies separate retention periods for raw, 5-minute, and 1-hour resolution data.

5. Challenge: Design a multi-cluster Prometheus architecture with Thanos that serves three data centers, provides 6 months of queryable data, and survives the loss of one entire data center.

What's Next

Your Prometheus knowledge is complete. Apply these patterns to build a production-grade monitoring platform for any infrastructure.

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.