SLOs, SLIs, and Error Budgets Explained — Practical Guide for Reliable Services

In this tutorial, you'll learn about SLOs, SLIs, and Error Budgets Explained. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Service Level Objectives (SLOs), Service Level Indicators (SLIs), and error budgets form the quantitative foundation of Site Reliability Engineering, replacing subjective "is the service stable?" judgments with measurable, data-driven reliability targets that balance feature velocity with system stability.

What You'll Learn

Why It Matters

Without SLOs, teams have no objective definition of "reliable enough." The infrastructure team tries to achieve 100% uptime, sacrificing all feature development. The product team ships features without regard for stability, causing frequent outages. SLOs resolve this conflict by establishing a shared contract: the service will be available X% of the time, and the remaining Y% (the error budget) can be spent on either reliability improvements or feature development. This gives teams permission to take calculated risks while maintaining accountability.

Real-World Use

DodaTech defines SLOs for all Durga Antivirus Pro services in a central YAML Repository. The API service has a 99.9% availability SLO with a 0.1% error budget (43 minutes per month). When the error budget is below 50%, deployment velocity slows -- features require additional manual testing. When above 50%, teams deploy freely. This system reduced customer-facing incidents by 70% while maintaining deployment frequency.

flowchart TD
    A["Service"] --> B["SLI: Request Latency"]
    A --> C["SLI: Error Rate"]
    A --> D["SLI: Availability"]
    B --> E["SLO: P99 < 500ms"]
    C --> F["SLO: < 0.1% errors"]
    D --> G["SLO: 99.9% uptime"]
    E --> H["Error Budget: 0.1%"]
    F --> H
    G --> H
    H --> I{"Budget Remaining?"}
    I -->|"> 50%"| J["Deploy freely"]
    I -->|"10%-50%"| K["Add manual QA"]
    I -->|"< 10%"| L["Stop deployments"]
    L --> M["Invest in reliability"]
    M --> H
    style E fill:#269539,color:#fff
    style F fill:#269539,color:#fff
    style G fill:#269539,color:#fff
    style L fill:#CC3333,color:#fff

Defining Good SLIs

A Service Level Indicator is a quantifiable measure of some aspect of service performance. Good SLIs measure what matters to users.

# slo-definitions.yaml
services:
  - name: api-gateway
    description: "Public API gateway for Durga Antivirus Pro"
    tier: "critical"
    slis:
      availability:
        type: "ratio"
        numerator: "sum(rate(http_requests_total{status!~'5..'}[5m]))"
        denominator: "sum(rate(http_requests_total[5m]))"
      latency_p99:
        type: "latency"
        query: "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))"
      latency_p50:
        type: "latency"
        query: "histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))"
    slos:
      - name: "availability-slo"
        sli: "availability"
        target: 0.999  # 99.9%
        window: "30d"
        description: "At most 43 minutes of downtime per month"
      - name: "latency-slo"
        sli: "latency_p99"
        target: 0.5  # 500ms
        comparator: "lt"
        window: "7d"

Error Budget Calculation

The error budget is the maximum acceptable failure over the SLO window. It is calculated as (1 - SLO_target) * total_events.

# error_budget.yml
# Service: API Gateway
# SLO target: 99.9% over 30 days
# Total requests in 30 days: 50,000,000
# Error budget: (1 - 0.999) * 50,000,000 = 50,000 failed requests

# Current state:
# Failed requests in current window: 12,345
# Budget consumed: 12,345 / 50,000 = 24.7%
# Budget remaining: 75.3%
# Status: SAFE (>50% remaining)

Expected behavior: After 30 days, if the service had more than 50,000 failed requests, the SLO was breached. The team reviews what caused the budget exhaustion and prioritizes reliability work.

Burn Rate Alerting

Burn rate alerts fire when the error budget is being consumed faster than expected. A burn rate of 1 means budget is consumed exactly on track to exhaust by the end of the window. A rate of 2 means it will exhaust in half the window.

# Prometheus burn rate alerts
groups:
  - name: slo_alerts
    rules:
      # 10x burn rate over 1 hour
      - alert: HighErrorBudgetBurnRate
        expr: |
          (
            1 - (
              sum(rate(http_requests_total{status=~"5.."}[1h]))
              / sum(rate(http_requests_total[1h]))
            )
          ) < 0.99
        for: 5m
        labels:
          severity: critical
          slo: "99.9%"
        annotations:
          summary: "High burn rate on API Gateway availability SLO"
          description: >
            Error budget is burning at >10x rate over the last hour.
            Current availability: {{ $value | humanizePercentage }}.
            SLO target: 99.9%.

      # 2x burn rate over 6 hours
      - alert: SloBurnRateWarning
        expr: |
          (
            1 - (
              sum(rate(http_requests_total{status=~"5.."}[6h]))
              / sum(rate(http_requests_total[6h]))
            )
          ) < 0.995
        for: 15m
        labels:
          severity: warning
          slo: "99.9%"
        annotations:
          summary: "Elevated burn rate on API Gateway availability SLO"
          description: >
            Error budget burning at >2x rate over 6 hours.
            Current availability: {{ $value | humanizePercentage }}.

Multi-Window, Multi-Burn-Rate Alerts

The standard approach for production SLO alerting combines multiple Windows and burn rates to balance detection speed with false positive reduction.

# Multi-window burn rate alerting
- alert: SloBurnRateCritical
  expr: |
    (
      (
        1 - (
          sum(rate(http_requests_total{status=~"5.."}[1h]))
          / sum(rate(http_requests_total[1h]))
        )
      ) < 0.99
    )
    and
    (
      (
        1 - (
          sum(rate(http_requests_total{status=~"5.."}[5m]))
          / sum(rate(http_requests_total[5m]))
        )
      ) < 0.99
    )
  for: 2m

Expected behavior: The alert requires BOTH a high burn rate over 1 hour and a high burn rate over 5 minutes. This prevents false alerts from brief traffic spikes while ensuring rapid detection of sustained problems.

SLO-Based Dashboards

{
  "title": "SLO Dashboard",
  "panels": [
    {
      "title": "Error Budget Remaining (30d)",
      "type": "gauge",
      "fieldConfig": {
        "defaults": {
          "unit": "percent",
          "min": 0,
          "max": 100,
          "thresholds": {
            "steps": [
              {"color": "red", "value": null},
              {"color": "yellow", "value": 10},
              {"color": "green", "value": 50}
            ]
          }
        }
      },
      "targets": [
        {
          "expr": "max(slo:error_budget_remaining_ratio{service=\"api-gateway\"}) * 100]
        }
      ]
    },
    {
      "title": "SLO Burn Rate (1h/6h/24h)",
      "type": "stat",
      "targets": [
        {"expr": "slo:burn_rate_1h{service=\"api-gateway\}"},
        {"expr": "slo:burn_rate_6h{service=\"api-gateway\}"},
        {"expr": "slo:burn_rate_24h{service=\"api-gateway\}"}
      ]
    }
  ]
}

Common Errors

1. Setting SLO targets without understanding user expectations: A 99.99% SLO (1 hour downtime per year) sounds impressive but costs exponentially more than 99.9%. If users are satisfied with 99.9%, the extra cost and complexity of 99.99% is wasted. Define SLOs based on user requirements, not engineering pride.

2. Measuring SLIs from the infrastructure rather than the user perspective: A server-side availability measurement misses failures that occur between the user and the server. If the server is healthy but the CDN or load balancer is failing, your SLI reports 100% while users see errors. Measure from synthetic probes or real user monitoring.

3 Not excluding planned maintenance from SLO Windows: Planned downtime for upgrades should not count against the error budget. Configure your SLI to exclude maintenance Windows or use a separate SLO for planned vs unplanned downtime.

4. Alerting on SLO breach rather than burn rate: An alert that fires when the SLO is already breached is too late. Burn rate alerts fire when the error budget is being consumed too quickly, giving the team time to respond before the SLO is violated.

5. Too many SLOs in one dashboard: A single service should have 2-3 well-chosen SLOs (availability, latency, and one domain-specific indicator). Having 15 SLOs per service creates confusion about what actually matters. Each SLO should have a clear owner who is accountable for it.

Practice Questions

1. What is an error budget and how does it balance reliability and feature velocity? Answer: An error budget is the maximum acceptable failure over the SLO window, calculated as (1 - SLO) * total events. When the error budget is healthy, teams can deploy features freely. When the budget is exhausted, teams must stop deploying and invest in reliability. This creates a data-driven mechanism for balancing velocity and stability.

2. What is the difference between SLI, SLO, and SLA? Answer: SLI (Service Level Indicator) is a specific metric (e.g., request latency). SLO (Service Level Objective) is a target value for that metric (e.g., P99 latency < 500ms). SLA (Service Level Agreement) is a contractual commitment to customers, usually with financial penalties, that is stricter than the internal SLO.

3. Why is burn rate alerting better than threshold-based alerting for SLOs? Answer: Threshold-based alerts fire when availability drops below X%, which is reactive. Burn rate alerts detect how quickly the error budget is being consumed, which is proactive. A 10x burn rate alert fires when the budget will exhaust in 3 days, giving time to respond before the SLO is breached.

4. How should error budget policy change based on the remaining budget? Answer: A common policy: above 50% remaining (green zone) -- deploy freely with standard CI/CD. 10-50% remaining (yellow zone) -- require additional manual testing and peer review for deployments. Below 10% (red zone) -- halt all deployments and focus on reliability improvements until the budget recovers.

Challenge

Define SLOs for a three-service platform (web frontend, API, database): identify the most important SLI for each service from a user perspective, set realistic SLO targets based on industry benchmarks, calculate the error budget for a 30-day window assuming 10M requests/month to the API, implement Prometheus burn rate alerts for 2x, 5x, and 10x burn rates with multi-window detection, create an error budget dashboard in Grafana with remaining budget gauge, burn rate stat panels, and SLI trend lines, and write an error budget policy document that defines what happens at each budget level.

Mini Project

Build a complete SLO management platform: create a YAML-based SLO definition Repository with SLI PromQL queries, SLO targets, and burn rate thresholds, write a Python script that generates Prometheus recording rules from the YAML definitions, deploy the recording rules to Prometheus, create multi-window burn rate alerts for 2x (warning) and 10x (critical) burn rates, build a Grafana dashboard with error budget remaining (threshold-colored gauge), burn rate chart (1h, 6h, 24h), SLI trend lines with SLO target overlay, and multi-service SLO comparison table, implement an error budget policy that posts to Slack when the budget drops below 50%, 25%, and 10%, and test the system by injecting errors and observing burn rate alerts and dashboard updates.

Related Resources

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.