Prometheus Recording Rules: Precomputed Metrics

In this tutorial, you'll learn about Prometheus Recording Rules: Precomputed Metrics. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Recording rules allow you to precompute frequently used or computationally expensive PromQL expressions in advance and store their results as new time series, making dashboards load faster and reducing query load on the Prometheus server.

What You'll Learn

In this tutorial, you will learn when to use recording rules, how to write them with proper naming conventions, and how to validate that precomputed metrics are working correctly in your dashboards.

Why It Matters

Complex PromQL queries with rate(), histogram_quantile(), and multiple aggregation steps can take seconds to evaluate, especially when querying months of data on busy servers. Every time a dashboard refreshes, every time an alert evaluates, these queries run again. Recording rules compute them once per scrape interval, making dashboards instant and reducing server CPU usage.

Real-World Use

DodaZIP processes millions of compression jobs daily. Its operations dashboard shows 95th percentile compression time, error rate by file type, and throughput per worker. All three metrics use recording rules. Without them, a dashboard refresh would run 15 expensive PromQL queries across 200 workers, taking over 10 seconds. With recording rules, the dashboard loads in under 1 second.

Step 1: Write Recording Rules

Create a recording rules file that precomputes common queries:

groups:
  - name: http_rates
    interval: 15s
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

      - record: instance:http_errors:rate5m
        expr: sum by (instance) (rate(http_requests_total{status=~"5.."}[5m]))

The record field sets the name of the new metric. Convention uses : as separators: level:metric:aggregation.

Step 2: Include in Prometheus Config

Add the recording rules file to <a href="/devops/prometheus-grafana/">Prometheus</a>.yml:

rule_files:
  - "rules.yml"
  - "recording_rules.yml"

Reload Prometheus. The new metrics appear in the web UI and can be queried like any other metric.

Step 3: Use in Dashboards

Instead of running the full PromQL expression, query the precomputed metric:

job:http_requests:rate5m

Expected output:

job:http_requests:rate5m{job="api"} 45.2
job:http_requests:rate5m{job="web"} 120.8
job:http_requests:rate5m{job="worker"} 33.1

The dashboard query returns instantly because the data is already computed.

When to Use Recording Rules

Use recording rules when:

• The same query appears in multiple dashboards or alerts
• The query takes more than 1 second to evaluate
• The query uses rate() or histogram_quantile() over large ranges
• You need to compute aggregates across hundreds of targets

Do not use recording rules for:

• Simple queries that are fast to evaluate
• Queries that change frequently (rule updates require reload)
• High-cardinality queries that would create too many new time series

Naming Convention

Follow the official Prometheus naming convention for recording rules:

level:metric:operation

Examples:

• node:node_cpu_seconds_total:rate5m -- CPU rate by node
• job:http_requests_total:sum_rate5m -- total request rate by job
• instance:memory_usage:avg -- average memory usage by instance

Common Mistakes

1. Creating Too Many Recording Rules

Each recording rule creates a new time series. Too many rules increase storage and memory usage. Only precompute queries that are actually expensive.

2. Wrong Aggregation Level

Recording rules that aggregate across all labels lose detail. If you later need per-instance data, create the rule at the correct granularity.

3. Forgetting the interval Field

Without interval, the rule evaluates at the global evaluation_interval. Set an explicit interval that matches or exceeds the scrape interval.

4. Not Using the Rule Naming Convention

Non-standard names cause confusion. Follow the level:metric:operation convention so other team members understand what each rule does.

5. Overlapping with Alerts

If an alert uses the same expression as a recording rule, the alert should query the recording rule result, not the original expression, to benefit from precomputation.

Practice Questions

1. What is the primary purpose of a recording rule? To precompute expensive PromQL expressions and store results as new time series, reducing query time for dashboards and alerts.

2. What naming convention should recording rules follow? level:metric:operation (e.g., job:http_requests:rate5m).

3. When should you NOT use a recording rule? For simple, fast queries or queries that change frequently.

4. What is the impact of creating too many recording rules? Each rule creates new time series, consuming additional disk space and memory on the Prometheus server.

5. Challenge: Identify three expensive queries in your current dashboards, write recording rules for them, and update the dashboards to use the precomputed metrics.

What's Next

Use service discovery to automatically find and scrape targets in dynamic environments like Kubernetes and AWS.

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.