Incident Response Runbooks and Automation — Structured Response for Production Outages

In this tutorial, you'll learn about Incident Response Runbooks and Automation. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Incident Response runbooks are structured, step-by-step documents that guide engineers through diagnosing and resolving production incidents, reducing mean-time-to-resolution by eliminating improvisation during high-stress situations.

What You'll Learn

Why It Matters

During a production outage, cognitive load spikes. Engineers under stress forget commands, skip diagnostic steps, and make decisions that worsen the situation. Runbooks replace panic with procedure -- a documented checklist that guides the engineer from symptom identification through diagnosis to resolution. Teams with well-maintained runbooks resolve incidents 3x faster than those without, and post-incident reviews consistently show that runbook-driven responses have fewer gaps and human errors.

Real-World Use

DodaTech maintains 40+ runbooks for Durga Antivirus Pro's production environment in a Git repository, rendered as a static site via MkDocs. When an alert fires, the Alertmanager notification includes a link to the relevant runbook. Automated runbooks (via StackStorm and Rundeck) handle 60% of common incidents -- auto-scaling failures, certificate renewals, and disk space alerts -- without human intervention.

flowchart TD
    A["Alert Fires"] --> B{"Runbook exists?"}
    B -->|"Yes: Automated"| C["Auto-Remediation"]
    B -->|"Yes: Manual"| D["On-Call Paged"]
    B -->|"No"| E["Improvise (slow)"]
    C --> F{"Remediation successful?"}
    F -->|"Yes"| G["Auto-close alert"]
    F -->|"No"| D
    D --> H["Open Runbook"]
    H --> I["Step 1: Verify alert"]
    I --> J["Step 2: Gather diagnostics"]
    J --> K["Step 3: Identify root cause"]
    K --> L["Step 4: Apply remediation"]
    L --> M["Step 5: Verify resolution"]
    M --> N["Step 6: Document timeline"]
    N --> O["Postmortem"]
    style C fill:#269539,color:#fff
    style D fill:#326CE5,color:#fff
    style L fill:#269539,color:#fff

Runbook Structure

Every runbook follows a consistent template so engineers can jump into any runbook and find the information they need immediately.

# runbooks/api-high-error-rate.md
---
title: API High Error Rate
severity: critical
services: [api-gateway, auth-api]
tags: [error-rate, backend]
last_updated: 2026-06-22
---

## Alert Description
The API error rate has exceeded the 5% threshold for 5 minutes.

## Symptoms
- Users report 500 errors in the web dashboard
- Alertmanager fires "HighErrorRate" alert
- Error rate graph shows a sudden or gradual increase in 5xx responses

## Pre-Checklist
- [ ] Verify alert is not a test or maintenance window
- [ ] Acknowledge the incident in PagerDuty
- [ ] Post in #incidents channel with alert details
- [ ] Declare severity level (SEV1, SEV2, SEV3)

## Diagnostic Steps

### 1. Check recent deployments
```bash
# Check if a recent deployment correlates with the error spike
kubectl get events --sort-by='.lastTimestamp' | tail -20

2. Identify failing endpoints

# Query <a href="/devops/prometheus-grafana/">Prometheus</a> for error rate by endpoint
# Expected: which endpoint has the highest error rate?
# Use <a href="/devops/prometheus-grafana/">Grafana</a> dashboard: API Error Rate by Endpoint

3. Check upstream dependencies

# Verify database connectivity
nc -zv postgres-primary 5432

# Check Redis
redis-cli -h redis-service ping

4. Examine application logs

# Query Loki for recent errors
# LogQL: {service="api-gateway", level="error"} |= "trace_id"

5. Check resource utilization

kubectl top pods -n production | grep api

Remediation Steps

Option A: Rollback recent deployment

# Rollback to previous version
kubectl rollout undo deployment/api-v2 -n production
kubectl rollout status deployment/api-v2 -n production

Option B: Scale up

# Increase replicas to handle load
kubectl scale deployment/api-v2 --replicas=10 -n production

Option C: Restart unhealthy pods

# Force restart without downtime
kubectl rollout restart deployment/api-v2 -n production

Verification

• Error rate drops below 1%
• No errors in Loki for the past 5 minutes
• All endpoints return HTTP 200
• PagerDuty alert resolves

Escalation

If the above steps do not resolve the issue within 15 minutes:

• Escalate to the backend team lead
• Engage the database administrator
• Notify the engineering director

Post-Incident

• Create a timeline of events
• Identify root cause
• File a follow-up ticket for permanent fix
• Update this runbook with lessons learned

## Automated Runbook with StackStorm

```yaml
# /opt/stackstorm/packs/auto_remediation/rules/disk_space_full.yaml
name: "auto_remediate_disk_space"
description: "Automatically clear disk space when usage exceeds 90%"
enabled: true
criteria:
  trigger:
    type: "prometheus"
    parameters:
      alertname: "DiskSpaceFull"
  conditions:
    - type: "regex"
      parameters:
        pattern: ".*instance: (.+) has less than 10%.*"
trigger:
  type: "prometheus.webhook"
  parameters:
    url: "http://stackstorm:9101/v1/webhooks/prometheus"
action:
  ref: "auto_remediation.cleanup_disk"
  parameters:
    host: "{{ trigger.body.instance }}"
    threshold: 90

# actions/cleanup_disk.py
from st2common.runners.base_action import Action
import subprocess

class CleanupDiskAction(Action):
    def run(self, host, threshold):
        commands = [
            "docker system prune -af --volumes 2>/dev/null",
            "journalctl --vacuum-time=3d 2>/dev/null",
            "find /var/log -name '*.gz' -delete 2>/dev/null",
            "find /tmp -type f -atime +7 -delete 2>/dev/null",
        ]
        results = []
        for cmd in commands:
            result = subprocess.run(
                ["ssh", host, cmd],
                capture_output=True, text=True, timeout=60
            )
            results.append({
                "command": cmd,
                "stdout": result.stdout.strip(),
                "stderr": result.stderr.strip(),
                "returncode": result.returncode,
            })
        return results

Expected behavior: When a "DiskSpaceFull" alert fires, StackStorm receives the webhook, runs the cleanup_disk action on the affected host, and if the disk usage drops below 90%, automatically resolves the alert. If cleanup fails or usage remains high, the alert continues and pages the on-call engineer.

Postmortem Template

# Incident Postmortem: INC-2026-06-22

## Summary
Brief description of what happened and impact.

## Timeline
| Time (UTC) | Event |
|------------|-------|
| 14:32 | Error rate spike detected by Prometheus |
| 14:33 | Alertmanager pages on-call engineer |
| 14:34 | Incident declared (SEV2) |
| 14:35 | Engineer acknowledges and opens runbook |
| 14:37 | Deployment rollback initiated |
| 14:42 | Error rate returns to baseline |
| 14:45 | Incident resolved |

## Root Cause
A configuration change deployed at 14:30 introduced an incorrect database
connection pool size that exhausted database connections.

## Contributing Factors
- The change was deployed without a review by the database team
- No staging environment test for connection pool limits
- The runbook did not include database connection pool diagnostics

## Action Items
| Action | Owner | Due Date | Ticket |
|--------|-------|----------|--------|
| Add connection pool monitoring | Backend team | 2026-06-29 | INC-2026-22-001 |
| Update CI/CD to require DB team review for pool changes | Platform team | 2026-06-29 | INC-2026-22-002 |
| Add connection pool diagnostic to runbook | On-call rotation | 2026-06-25 | INC-2026-22-003 |
| Test connection pool limits in staging | QA team | 2026-06-30 | INC-2026-22-004 |

## What Went Well
- Alerting detected the issue within 1 minute
- Rollback procedure was fast and reliable
- Communication in the incident channel was clear

## What Went Wrong
- The change bypassed the required review process
- The runbook did not cover database connection pool diagnostics
- No monitoring was in place for database connection utilization

## Metrics
- Time to detection: 1 minute
- Time to response: 4 minutes
- Time to resolution: 10 minutes
- Total impact duration: 13 minutes
- Users affected: ~2,500

Common Errors

1. Runbooks not updated after incidents: The most common failure mode is a runbook that describes a fix that no longer works because the system architecture changed. Every postmortem MUST include a runbook update action item. Outdated runbooks are worse than no runbook because they waste time during an incident.

2. Skipping verification steps: After applying a remediation, engineers often assume it worked without verifying. Every runbook must include explicit verification steps (check metrics, test endpoints, confirm in logs) before marking the incident as resolved.

3. No automated fallback for failed remediation: Automated runbooks that fail silently are dangerous. If the auto-remediation action fails, it must escalate to a human. A failed docker system prune because the disk is full should trigger a page, not a silent failure.

4. Manual runbooks that require jumping between tools: A runbook that says "check metric X in Grafana, then query logs in Kibana, then restart service in Kubernetes Console" forces the engineer to context-switch across 3 tools. Provide direct links to the relevant dashboard, log query, and kubectl command.

5. Postmortems without actionable follow-up: A postmortem that documents what happened but does not assign owners or create tickets is a waste of effort. Every action item must have an owner and a due date. Blameless postmortems are about systemic improvements, not documenting failures.

Practice Questions

1. What is the difference between a manual runbook and an automated runbook? Answer: A manual runbook documents step-by-step instructions for a human to follow. An automated runbook executes those steps programmatically via tools like StackStorm, Rundeck, or a CI/CD pipeline. Automated runbooks handle common incidents without human intervention, while manual runbooks guide humans through rare or complex incidents.

2. Why should runbooks be stored in version control? Answer: Version control provides an audit trail of changes, enables Pull Request reviews for accuracy, allows linking incidents to specific runbook versions, and enables automated rendering via CI/CD (MkDocs, Hugo, or Sphinx to HTML).

3. What is the purpose of a pre-mortem or game day exercise? Answer: Game days simulate incidents to test runbooks in a controlled environment. They reveal gaps in runbook coverage, incorrect steps, or missing permissions long before a real incident occurs. Teams should run game days quarterly and update runbooks based on findings.

4. How does severity classification affect Incident Response? Answer: Severity determines response speed, escalation path, and notification audience. SEV1 (critical, customer-facing outage) pages the entire on-call chain and the engineering director. SEV3 (minor, internal impact) creates a ticket for next-business-day resolution. Clear severity definitions prevent over-escalation of minor issues and under-response to critical ones.

Challenge

Create a complete Incident Response runbook suite for an e-commerce platform: write runbooks for 5 common scenarios (high error rate, database connection exhaustion, certificate expiry, deployment failure, and slow page load), define severity levels (SEV1-SEV4) with clear criteria, implement one automated runbook using StackStorm or a Shell Script that handles certificate renewal automatically, create a postmortem template with timeline, root cause, action items, and "what went well/what went wrong" sections, and conduct a tabletop exercise where you simulate an incident and walk through the runbook with a team.

Mini Project

Build a complete Incident Response platform: set up a Git repository with MkDocs for rendering markdown runbooks as a searchable static site, write 10 runbooks covering common infrastructure and application incidents, implement a CI/CD pipeline that rebuilds and deploys the runbook site when a Pull Request is merged, configure Alertmanager to include links to relevant runbooks in alert notifications (using alertmanager-<a href="/backend/webhooks/">webhook</a>-relay or custom templates), set up StackStorm with 3 automated runbooks (disk cleanup, certificate renewal, dead Pod restart), integrate with PagerDuty for on-call scheduling and Slack for incident communication, create a severity matrix with clear definitions and escalation paths, and run a game day simulation where you trigger a test incident (e.g., kill a critical Pod) and walk through the runbook with the team, timing each step and identifying gaps.

Related Resources

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.