Cron Anomaly Detection — Automated Monitoring for Unusual Patterns

In this tutorial, you will learn about Cron Anomaly Detection. We cover key concepts, practical examples, and best practices to help you master this topic.

Learn cron-based anomaly detection: schedule automated analysis of operational metrics to detect spikes, drops, and outliers, compare against historical baselines computed by Cron Jobs, and trigger alerts for detected anomalies.

What You Learn

You will learn how to use cron for anomaly detection: computing statistical baselines from historical data, detecting outliers in metrics, identifying trend changes, and alerting on anomalous patterns automatically.

Why It Matters

Static thresholds miss anomalies that deviate from normal patterns in unexpected ways. A 20% increase in error rate might be normal during a traffic spike but anomalous during normal hours. Cron-based anomaly detection learns normal patterns and alerts on deviations.

Real-World Use

DodaTech runs anomaly detection cron jobs every 15 minutes. The jobs analyze 50+ metrics across all services, comparing current values against rolling 7-day baselines. When the payment service error rate spiked 3x above baseline at 3 AM, the cron job detected it and paged on-call within 1 minute.

Statistical Baseline Computation

import time
import random
import statistics
from datetime import datetime, timedelta

class BaselineComputer:
    def __init__(self, metric_name, window_days=7):
        self.metric_name = metric_name
        self.window_days = window_days
        self.history = []

    def add_value(self, value):
        self.history.append({
            'value': value,
            'timestamp': datetime.now()
        })
        cutoff = datetime.now() - timedelta(days=self.window_days)
        self.history = [h for h in self.history if h['timestamp'] > cutoff]

    def compute_baseline(self):
        if len(self.history) < 10:
            return None
        values = [h['value'] for h in self.history]
        return {
            'mean': statistics.mean(values),
            'stdev': statistics.stdev(values) if len(values) > 1 else 0,
            'p50': statistics.median(values),
            'p95': sorted(values)[int(len(values) * 0.95)],
            'p99': sorted(values)[int(len(values) * 0.99)],
            'sample_count': len(values),
        }

    def is_anomalous(self, value, threshold=3):
        baseline = self.compute_baseline()
        if not baseline or baseline['stdev'] == 0:
            return False
        z_score = abs(value - baseline['mean']) / baseline['stdev']
        is_anom = z_score > threshold
        if is_anom:
            print(f"ANOMALY: {self.metric_name} = {value} (z-score: {z_score:.1f}, mean: {baseline['mean']:.1f}, stdev: {baseline['stdev']:.1f})")
        return is_anom

baseline = BaselineComputer("api_latency_ms")
for i in range(100):
    baseline.add_value(random.gauss(200, 30))

baseline.compute_baseline()
baseline.is_anomalous(500)

Expected output:

ANOMALY: api_latency_ms = 500 (z-score: 10.0, mean: 200.0, stdev: 30.0)

Anomaly Detection Engine

import time
import random
from datetime import datetime

class AnomalyDetector:
    def __init__(self):
        self.metrics = {}

    def add_metric(self, name, window_days=7, threshold=3):
        self.metrics[name] = {
            'baseline': BaselineComputer(name, window_days),
            'threshold': threshold,
            'alerts': 0
        }

    def check_metric(self, name, value):
        if name not in self.metrics:
            return None
        metric = self.metrics[name]
        metric['baseline'].add_value(value)
        is_anom = metric['baseline'].is_anomalous(value, metric['threshold'])
        if is_anom:
            metric['alerts'] += 1
            return {'metric': name, 'value': value, 'anomalous': True}
        return {'metric': name, 'value': value, 'anomalous': False}

    def get_summary(self):
        total = len(self.metrics)
        alerts = sum(m['alerts'] for m in self.metrics.values())
        return {'metrics': total, 'current_alerts': alerts}

detector = AnomalyDetector()
detector.add_metric("error_rate", window_days=7, threshold=3)
detector.add_metric("request_count", window_days=7, threshold=3)

for i in range(50):
    detector.check_metric("error_rate", random.gauss(2, 1))
    detector.check_metric("request_count", random.gauss(1000, 100))

detector.check_metric("error_rate", 15)
print(detector.get_summary())

Expected output:

ANOMALY: error_rate = 15 (z-score: 13.0, mean: 2.0, stdev: 1.0)
{'metrics': 2, 'current_alerts': 1}

Common Mistakes

1. No Historical Baseline

Without a baseline, you cannot detect anomalies. Collect at least 7 days of data to establish normal patterns. Consider seasonal patterns: compare Monday 3 PM against other Mondays, not against Sunday 3 AM.

2. Assuming Normal Distribution

Not all metrics follow a normal distribution. Request latency is often skewed (many fast, few slow). Error rates are binary. Use percentile-based thresholds (P95, P99) for non-normal distributions instead of mean/stdev.

3. Alerting on Every Anomaly

If you alert on every statistical outlier, you get too many alerts. Require: anomaly persists for N consecutive checks, or anomaly exceeds threshold by 2x the warning level. Tune threshold based on observed false positive rate.

4. No Time-of-Day Awareness

A metric value that is anomalous at 3 AM may be normal at 3 PM. Compute separate baselines per hour of day. Compare current value against the baseline for the same hour on previous days.

5. Ignoring Correlation

A spike in error rate is not anomalous if it correlates with a spike in traffic. Correlate metrics before alerting: if traffic is also up 3x, the error rate increase may be proportional rather than anomalous.

Practice Questions

1. How do you compute a baseline for anomaly detection?

Collect historical data over a window (7-30 days). For normally distributed data, compute mean and standard deviation. For non-normal data, use percentile thresholds (P95, P99). Adjust for time-of-day patterns.

2. What is a z-score and how is it used?

Z-score measures how many standard deviations a value is from the mean. A z-score > 3 means the value is more than 3 standard deviations from the norm, which is statistically unlikely (0.3% probability).

3. How do you handle false positives in anomaly detection?

Require anomaly to persist for 2-3 consecutive checks before alerting. Use a higher threshold during known volatile periods. Implement alert suppression for correlated anomalies (if one metric is anomalous, suppress related metrics).

4. How do you detect slow-burn anomalies (gradual degradation)?

Use trend detection: compute the slope of the metric over time (linear regression). Alert on significant slope changes even if individual values are within normal range. Example: memory growing 1% per day for 30 days.

Challenge

Build an anomaly detection system with cron: (1) every 15 minutes: collect 20+ metrics (error rates, latencies, request counts, resource usage), (2) baseline computation: 7-day rolling window with time-of-day segmentation (24 separate baselines), (3) anomaly detection: z-score (normal data), percentile (non-normal data), trend detection (gradual changes), (4) correlation engine: if error rate spikes AND traffic is normal -> alert, if both spike -> log only, (5) alert suppression: do not alert on same anomaly twice, suppress related metrics, (6) dashboard: metrics timeline with anomaly markers, baseline overlay, current status.

FAQ

Mini Project: Anomaly Detection System

Build a cron-based anomaly detection system: (1) metric collector: runs every 15 minutes, collects 20 metrics from all services via API endpoints, (2) baseline computer: rolling 7-day window, time-of-day segmentation (24 baselines per metric), mean/stdev for normal data, P95/P99 for non-normal, (3) anomaly detector: z-score for normal, percentile for non-normal, trend detection for gradual changes (linear regression slope), (4) correlator: suppress alerts when related metrics explain the anomaly (e.g., error rate increase correlated with traffic increase), (5) alert rules: anomaly >3-sigma for 2 consecutive checks, trend slope >5% per day for 7 days, (6) dashboard: metric timeline with anomaly highlights, baseline overlay, alert history, (7) feedback loop: allow marking anomalies as false positives to tune thresholds.

What's Next

Now that you understand anomaly detection with cron, explore cost optimization with cron, then learn about workflow automation.