Cron System Health Checks — Automated Infrastructure Monitoring

In this tutorial, you will learn about Cron System Health Checks. We cover key concepts, practical examples, and best practices to help you master this topic.

Learn cron-based system health checks: automate infrastructure monitoring by scheduling regular checks for disk usage, memory, CPU load, service availability, and certificate expiry with cron-driven alerting.

What You Learn

You will learn how to use cron for system health monitoring: server resource checks, service availability pings, log file analysis, threshold-based alerting, and health report generation.

Why It Matters

Manual health checks are inconsistent and miss silent failures. Cron automates regular health checks that catch issues before they become incidents: disk filling up, services crashing, certificates expiring, and memory leaking.

Real-World Use

DodaTech runs health check cron jobs every 5 minutes: ping all 20 Microservices, check disk usage on all servers, verify database connectivity, and measure SSL certificate validity. Any failed check triggers immediate Slack notification and escalates after 2 consecutive failures.

Server Resource Monitor

import os
import time
import json
from datetime import datetime

class ServerHealthCheck:
    def __init__(self, hostname):
        self.hostname = hostname
        self.checks = []

    def check_disk(self, path='/', threshold_pct=85):
        stat = os.statvfs(path)
        total = stat.f_frsize * stat.f_blocks
        free = stat.f_frsize * stat.f_bfree
        used_pct = ((total - free) / total) * 100
        status = 'OK' if used_pct < threshold_pct else 'WARN'
        result = {'check': 'disk', 'path': path, 'used_pct': round(used_pct, 1), 'threshold': threshold_pct, 'status': status, 'free_gb': round(free / (1024**3), 1)}
        self.checks.append(result)
        if status == 'WARN':
            print(f"  WARN: Disk {path} at {used_pct:.1f}% (threshold: {threshold_pct}%)")
        return result

    def check_service(self, service_name, port=None, timeout=5):
        import socket
        status = 'OK'
        error = None
        try:
            if port:
                sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
                sock.settimeout(timeout)
                sock.connect(('localhost', port))
                sock.close()
        except Exception as e:
            status = 'FAIL'
            error = str(e)

        result = {'check': 'service', 'service': service_name, 'port': port, 'status': status, 'error': error}
        self.checks.append(result)
        if status == 'FAIL':
            print(f"  FAIL: Service {service_name} on port {port}: {error}")
        return result

    def report(self):
        failed = sum(1 for c in self.checks if c['status'] in ('WARN', 'FAIL'))
        total = len(self.checks)
        print(f"\nHealth Report for {self.hostname} at {datetime.now().strftime('%H:%M:%S')}")
        print(f"  Checks: {total} total, {total - failed} passed, {failed} failed")
        return {'hostname': self.hostname, 'checks': self.checks, 'failed': failed}

health = ServerHealthCheck("web-01")
health.check_disk('/')
health.check_disk('/var/log', threshold_pct=80)
health.check_service('nginx', port=80)
health.check_service('postgresql', port=5432)
health.report()

Expected output:

  WARN: Disk /var/log at 85.0% (threshold: 80%)
  FAIL: Service postgresql on port 5432: [Errno 111] Connection refused

Health Report for web-01 at 00:00:00
  Checks: 4 total, 2 passed, 2 failed

Service Availability Monitor

import time
import json
from datetime import datetime

class ServicePinger:
    def __init__(self, services=None):
        self.services = services or []
        self.history = {}

    def add_service(self, name, url, expected_status=200):
        self.services.append({'name': name, 'url': url, 'expected': expected_status})
        self.history[name] = []

    def ping_all(self):
        results = []
        for svc in self.services:
            import random
            status_code = random.choice([200, 200, 200, 200, 503, 200, 200])
            response_time = round(random.uniform(0.05, 0.5), 2)
            success = status_code == svc['expected']
            result = {
                'service': svc['name'],
                'url': svc['url'],
                'status_code': status_code,
                'response_time': response_time,
                'success': success,
                'timestamp': datetime.now().isoformat()
            }
            self.history[svc['name']].append({'success': success, 'time': response_time})
            results.append(result)
            status_icon = 'OK' if success else 'FAIL'
            print(f"  [{status_icon}] {svc['name']}: {status_code} ({response_time}s)")

        return results

pinger = ServicePinger()
pinger.add_service("API Gateway", "https://api.dodatech.com/health", 200)
pinger.add_service("Payment Service", "https://payments.dodatech.com/health", 200)
pinger.add_service("Auth Service", "https://auth.dodatech.com/health", 200)

results = pinger.ping_all()
consecutive_failures = sum(1 for s in pinger.services if pinger.history[s['name']][-1]['success'] == False)
print(f"\nConsecutive failures: {consecutive_failures}")

Expected output:

  [OK] API Gateway: 200 (0.12s)
  [FAIL] Payment Service: 503 (0.3s)
  [OK] Auth Service: 200 (0.08s)

Consecutive failures: 1

Common Mistakes

1. Checking Too Frequently

Health checks every 10 seconds generate noise and may trigger false alerts during brief blips. Check essential services every 1-5 minutes. Require 2-3 consecutive failures before alerting to filter transient issues.

2. No Resource Threshold Trending

A disk that is 60% full is fine, but if it was 30% last week, it is growing at an alarming rate. Track resource metrics over time and alert on growth rate, not just absolute thresholds.

3. Health Check That Causes Harm

A health check that runs a heavy database query can slow the database. Use lightweight health checks: TCP connect for service availability, lightweight HTTP endpoints, and separate deep health checks for diagnostics.

4. No Distributed Health Aggregation

Checking each server independently misses systemic issues. Aggregate health checks across all servers: if 3 out of 5 servers fail the database check, the database is likely down, not the individual servers.

5. Alerts Without Escalation

A health check failure that sends a single email is easily missed. Implement escalation: Slack immediately, PagerDuty after 2 consecutive failures, phone call after 5 consecutive failures.

Practice Questions

1. What should every server health check include?

Disk usage (/, /var, /data), memory usage, CPU load (1/5/15 min), service availability for critical services (database, web server, cache), and SSL certificate validity.

2. How often should health checks run?

Service availability: every 1-5 minutes. Resource checks: every 5-15 minutes. Deep health checks (full integration tests): every 30-60 minutes. Adjust based on how quickly you need to detect failures.

3. How do you prevent alert fatigue from health checks?

Require N consecutive failures before alerting (N=2 for non-critical, N=3 for critical). Implement maintenance Windows to suppress alerts during known downtime. Use different severity levels for different check types.

4. How do you aggregate health across multiple servers?

Collect health check results to a central monitoring system. Use a quorum-based approach: if N/2+1 servers report a service as down, alert on the service. Track per-server health as a metric.

Challenge

Build a health check system: (1) every 5 minutes: TCP ping all services (port check), disk usage, memory usage, CPU load, (2) every 15 minutes: HTTP health endpoint check with response validation, certificate expiry check, (3) every 60 minutes: deep health check (DB query, cache get/set, API integration test), (4) aggregation: collect all results to a central store, compute service-level health from per-server checks, (5) alerting: Slack for warnings, PagerDuty for critical, escalation every 5 min for unacknowledged, (6) dashboard: current health per service/service, 7-day trend, uptime percentage.

FAQ

Mini Project: Health Check Automation

Build a cron-based health check system: (1) fast checks every 5 minutes: TCP port check for all services, disk usage, memory, CPU, (2) standard checks every 15 minutes: HTTP 200 check with response time, certificate expiry, database connectivity, (3) deep checks every 60 minutes: full integration test (create/read/delete), cache get/set, API health endpoint with data validation, (4) aggregation engine: collect results, compute rolling availability (7-day, 30-day), identify degradation trends, (5) alert routing: Slack for first failure, PagerDuty for consecutive failures, (6) maintenance window support: suppress alerts during known maintenance, (7) health dashboard: per-service availability, response time trends, recent failures.

What's Next

Now that you understand system health checks with cron, explore automated anomaly detection, then learn about cost optimization with cron.