API Monitoring: Production Health Checks and Performance Tracking
In this tutorial, you will learn about API Monitoring: Production Health Checks and Performance Tracking. We cover key concepts, practical examples, and best practices to help you master this topic.
API monitoring continuously checks production API health through synthetic probes, health endpoints, uptime checks, response time tracking, anomaly detection, and integrated dashboards for real-time visibility.
What You'll Learn
How to monitor APIs in production: synthetic monitoring with automated probes, health check endpoint design, uptime and latency tracking, alerting on anomalies and regressions, distributed tracing with OpenTelemetry, and integrating with Grafana/Datadog dashboards.
Why It Matters
Bugs reach production despite testing. Monitoring detects issues before users report them. DodaTech monitors 50+ API endpoints every 30 seconds, detecting 90% of incidents before customer impact.
Real-World Use
A deployment introduces a 2-second latency regression in the checkout API. DodaTech's synthetic monitor detects the increase from 200ms to 2s, triggers a PagerDuty alert, and auto-rolls back the deployment — all within 3 minutes.
flowchart LR
A["Synthetic\nMonitor (30s)"] --> B["API Health\nCheck"]
B --> C{"Response\nOK?"}
C -->|Yes| D["Check\nLatency"]
C -->|No| E["Alert:\nDown"]
D --> F{"Avg < 500ms?"}
F -->|Yes| G["Healthy"]
F -->|No| H["Alert:\nSlow"]
E --> I["Incident\nResponse"]
H --> I
style A fill:#dbeafe,stroke:#2563eb
style E fill:#fecaca,stroke:#dc2626
style H fill:#fef3c7,stroke:#d97706
style I fill:#fecaca,stroke:#dc2626
Health Check Endpoint
# FastAPI health check endpoint
from fastapi import FastAPI, APIRouter
from pydantic import BaseModel
import time
import os
router = APIRouter()
class HealthStatus(BaseModel):
status: str
version: str
uptime: float
database: str
dependencies: dict
timestamp: float
START_TIME = time.time()
@router.get("/health", response_model=HealthStatus)
async def health_check():
# Check database connectivity
db_status = "healthy"
try:
await db.execute("SELECT 1")
except Exception as e:
db_status = f"unhealthy: {str(e)}"
# Check external dependencies
deps = {
"stripe": await check_external("https://api.stripe.com"),
"twilio": await check_external("https://api.twilio.com"),
"redis": await check_redis()
}
overall = "healthy" if all(d == "healthy" for d in deps.values()) else "degraded"
return HealthStatus(
status=overall,
version=os.getenv("APP_VERSION", "unknown"),
uptime=time.time() - START_TIME,
database=db_status,
dependencies=deps,
timestamp=time.time()
)
# Expected response:
# {
# "status": "healthy",
# "version": "2.3.1",
# "uptime": 864000.0,
# "database": "healthy",
# "dependencies": {
# "stripe": "healthy",
# "twilio": "healthy",
# "redis": "healthy"
# },
# "timestamp": 1719561600.0
# }
Synthetic Monitoring Script
# synthetic_monitor.py - runs every 30-60 seconds
import requests
import time
import json
from datetime import datetime
MONITORED_ENDPOINTS = [
{"name": "Health Check", "method": "GET", "url": "/health", "expected": 200},
{"name": "List Users", "method": "GET", "url": "/users?limit=1", "expected": 200},
{"name": "Create Order", "method": "POST", "url": "/orders",
"body": {"product_id": "prod-monitor", "quantity": 1}, "expected": 201},
]
def run_monitoring_cycle(base_url, api_key):
results = []
for endpoint in MONITORED_ENDPOINTS:
start = time.time()
try:
headers = {"Authorization": f"Bearer {api_key}"}
if endpoint["method"] == "GET":
resp = requests.get(
f"{base_url}{endpoint['url']}",
headers=headers, timeout=5
)
else:
resp = requests.post(
f"{base_url}{endpoint['url']}",
json=endpoint.get("body"),
headers=headers, timeout=5
)
duration = (time.time() - start) * 1000
result = {
"endpoint": endpoint["name"],
"status": "pass" if resp.status_code == endpoint["expected"] else "fail",
"http_status": resp.status_code,
"duration_ms": round(duration, 2),
"timestamp": datetime.utcnow().isoformat()
}
except requests.Timeout:
result = {
"endpoint": endpoint["name"],
"status": "fail",
"error": "timeout",
"duration_ms": 5000,
"timestamp": datetime.utcnow().isoformat()
}
except Exception as e:
result = {
"endpoint": endpoint["name"],
"status": "fail",
"error": str(e),
"duration_ms": 0,
"timestamp": datetime.utcnow().isoformat()
}
results.append(result)
print(f" {result['endpoint']}: {result['status']} ({result.get('duration_ms', 0)}ms)")
return results
# results = run_monitoring_cycle("https://api.dodatech.com/v1", "monitor-key")
# Expected output:
# Health Check: pass (45ms)
# List Users: pass (120ms)
# Create Order: pass (230ms)
Prometheus Metrics Integration
# Expose metrics for Prometheus scraping
from prometheus_client import Counter, Histogram, Gauge, generate_latest
from fastapi import Response
# Define metrics
http_requests_total = Counter(
'http_requests_total',
'Total HTTP requests',
['method', 'endpoint', 'status']
)
http_request_duration = Histogram(
'http_request_duration_seconds',
'HTTP request duration in seconds',
['method', 'endpoint'],
buckets=[0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0]
)
active_users = Gauge('active_users', 'Number of active users')
# Update metrics in middleware or route handlers
@app.middleware("http")
async def monitor_requests(request, call_next):
method = request.method
endpoint = request.url.path
start = time.time()
response = await call_next(request)
duration = time.time() - start
http_requests_total.labels(
method=method, endpoint=endpoint,
status=response.status_code
).inc()
http_request_duration.labels(
method=method, endpoint=endpoint
).observe(duration)
return response
# Metrics endpoint for Prometheus scraping
@router.get("/metrics")
async def metrics():
return Response(generate_latest(), media_type="text/plain")
# Expected Prometheus output:
# HELP http_requests_total Total HTTP requests
# TYPE http_requests_total counter
# http_requests_total{method="GET",endpoint="/users",status="200"} 1024
# http_requests_total{method="POST",endpoint="/orders",status="201"} 512
# HELP http_request_duration_seconds HTTP request duration in seconds
# TYPE http_request_duration_seconds histogram
# http_request_duration_seconds_bucket{method="GET",endpoint="/users",le="0.01"} 100
Alerting Rules
# prometheus-alerts.yml
groups:
- name: api_alerts
rules:
- alert: APIHighErrorRate
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
> 0.05
for: 5m
labels:
severity: critical
annotations:
summary: "API error rate above 5%"
description: "Error rate is {{ $value | humanizePercentage }}"
- alert: APIHighLatency
expr: |
histogram_quantile(0.95,
rate(http_request_duration_seconds_bucket[5m])
) > 1.0
for: 5m
labels:
severity: warning
annotations:
summary: "API p95 latency above 1 second"
- alert: APIDown
expr: |
up{job="dodatech-api"} == 0
for: 1m
labels:
severity: critical
annotations:
summary: "API instance is down"
Common Mistakes
1. Monitoring Only the Health Endpoint
The /health endpoint often works when other endpoints fail. Monitor critical business endpoints (create order, list products, Process payment) not just the health check.
2. No Synthetic Data Cleanup
Monitors that create test orders, users, or products must clean up after themselves. Schedule periodic cleanup jobs or use monitoring-only test accounts with auto-expiry.
3. Alert Fatigue from Noisy Alerts
Alerting on every 5xx or latency spike causes ignored alerts. Set proper thresholds, duration (alert after 5 minutes, not 1), and routing (warning vs critical) to reduce noise.
4. Not Monitoring Dependencies
Your API may be healthy but a downstream service (database, Stripe, Redis) may be degraded. Monitor dependency health in the health endpoint and set up dependency-specific alerts.
5. No SLA Tracking
Without tracking uptime against SLA targets, you don't know if you're meeting commitments. Track monthly uptime %, p95/p99 latency, and error budget consumption.
Practice Questions
- What is synthetic monitoring and why is it important?
- What should a health check endpoint include?
- How do you set up alerting for API latency regressions?
- What is the difference between active and passive monitoring?
Answers:
- Synthetic monitoring runs automated probes against API endpoints at regular intervals (30-60s). It detects issues before users report them, works even with zero user traffic, and validates functionality not just uptime.
- Service status, version, uptime, database connectivity, dependency health (external APIs, Redis, database), and timestamp. Each dependency should be checked individually with clear pass/fail status.
- Track p95/p99 latency with Prometheus histograms. Set alert:
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 1.0for 5 minutes. Alert on trend increase, not single spikes. - Active monitoring sends synthetic requests to probe API health. Passive monitoring observes real user traffic (APM, RUM) to detect issues. Active catches problems before users notice; passive catches user-specific issues.
Challenge: Build a complete API monitoring system: health check endpoint with 3 dependency checks, Prometheus metrics for request count and latency, Grafana dashboard with latency heatmap and error rate panels, synthetic monitor script for 3 critical endpoints, PagerDuty alerting for high error rate and latency, and monthly SLA report.
FAQ
Mini Project
Build a complete API monitoring stack: FastAPI health endpoint with dependency checks, Prometheus metrics for latency/error rate, Grafana dashboard with 6 panels, synthetic monitor Python script for 3 endpoints, Prometheus alerting rules for error rate and latency, PagerDuty integration, and monthly SLA report generator.
What's Next
Test Automation Framework — build a reusable API test automation framework.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro