Cron Scheduled Scraping — Automated Web Scraping with Cron

In this tutorial, you will learn about Cron Scheduled Scraping. We cover key concepts, practical examples, and best practices to help you master this topic.

Learn cron-based web scraping: schedule periodic data collection from external APIs and websites with cron, implement rate limit-aware scraping schedules, deduplicate collected data, and monitor scraping job health.

What You Learn

You will learn how to use cron for automated web scraping: scheduling periodic data collection, respecting API rate limits with cron timing, detecting and handling duplicate data, error handling for transient failures, and data quality monitoring.

Why It Matters

Manual data collection is slow and unreliable. Cron automates periodic data collection from APIs and websites, ensuring consistent data flow for analytics, monitoring, and business intelligence without human intervention.

Real-World Use

DodaTech's security team uses cron-based scraping to collect threat intelligence feeds every 15 minutes, competitor pricing data daily, and SSL certificate expiration data hourly. Cron ensures data is collected consistently regardless of holidays or weekends.

Rate-Limited Scraper

import time
import random
from datetime import datetime

class RateLimitedScraper:
    def __init__(self, requests_per_minute=10):
        self.min_interval = 60.0 / requests_per_minute
        self.last_request = 0

    def wait_if_needed(self):
        elapsed = time.time() - self.last_request
        if elapsed < self.min_interval:
            sleep_time = self.min_interval - elapsed
            time.sleep(sleep_time)

    def fetch(self, url):
        self.wait_if_needed()
        self.last_request = time.time()

        duration = random.uniform(0.1, 0.3)
        time.sleep(duration)

        status_code = 200 if random.random() > 0.1 else 429
        if status_code == 429:
            retry_after = random.randint(5, 30)
            print(f"  [429] {url} — rate limited, retry after {retry_after}s")
            return None

        print(f"  [200] {url} ({duration:.1f}s)")
        return {"url": url, "data": f"data_{int(time.time())}"}

class ScrapingCron:
    def __init__(self, name, schedule_minutes):
        self.name = name
        self.schedule = schedule_minutes
        self.scraper = RateLimitedScraper(requests_per_minute=10)
        self.urls = []

    def add_source(self, url):
        self.urls.append(url)

    def run(self):
        print(f"[{datetime.now().strftime('%H:%M:%S')}] Running {self.name}")
        results = []
        for url in self.urls:
            result = self.scraper.fetch(url)
            if result:
                results.append(result)
        print(f"  Collected {len(results)} items")
        return results

scraper = ScrapingCron("Threat Intel Feed", schedule_minutes=15)
scraper.add_source("https://api.threatintel.com/feeds/latest")
scraper.add_source("https://api.threatintel.com/feeds/malware")
scraper.run()

Expected output:

[00:00:00] Running Threat Intel Feed
  [200] https://api.threatintel.com/feeds/latest (0.2s)
  [200] https://api.threatintel.com/feeds/malware (0.1s)
  Collected 2 items

Deduplication

import hashlib
import json
from datetime import datetime

class ScrapedDataDeduplicator:
    def __init__(self):
        self.seen_hashes = set()

    def compute_hash(self, item):
        content = json.dumps(item, sort_keys=True)
        return hashlib.md5(content.encode()).hexdigest()

    def is_duplicate(self, item):
        item_hash = self.compute_hash(item)
        if item_hash in self.seen_hashes:
            return True
        self.seen_hashes.add(item_hash)
        return False

    def deduplicate(self, items):
        unique = []
        duplicates = 0
        for item in items:
            if self.is_duplicate(item):
                duplicates += 1
            else:
                unique.append(item)
        print(f"Dedup: {len(unique)} unique, {duplicates} duplicates removed")
        return unique

dedup = ScrapedDataDeduplicator()
items = [
    {"id": 1, "title": "Alert A", "severity": "high"},
    {"id": 2, "title": "Alert B", "severity": "medium"},
    {"id": 1, "title": "Alert A", "severity": "high"},
    {"id": 3, "title": "Alert C", "severity": "low"},
]

dedup.deduplicate(items)

Expected output:

Dedup: 3 unique, 1 duplicates removed

Common Mistakes

1. Ignoring Rate Limits

Sending requests faster than the API allows gets your IP blocked. Use cron to space requests: schedule scraping jobs at intervals that respect the API's rate limits. Implement client-side Rate Limiting as a safety net.

2. No Data Deduplication

Running a scraping cron job every 15 minutes produces 96 data points per day. Without deduplication, you store identical data multiple times, wasting storage and complicating analysis. Use content hashing to detect and skip duplicates.

3. Scraping During Target Maintenance

Scraping during the target's maintenance window produces error data. Check the target's status page or health endpoint before scraping. Skip runs during known maintenance Windows.

4. No Monitoring of Data Freshness

If scraping fails for 3 consecutive runs, data becomes increasingly stale. Monitor: time since last successful scrape, data point age, collection gap duration. Alert if data is older than 2x the collection interval.

5. Storing Raw Data Without Validation

A malformed API response or HTML structure change can corrupt your dataset. Validate scraped data against a schema before storing. Implement structure change detection: if field count changes significantly, alert for manual review.

Practice Questions

1. How do you rate-limit a cron scraping job?

Set the scraping interval to respect the API's rate limits. If the API allows 100 requests per minute, schedule the cron job to run at most every 60/100 = 0.6 seconds per request. Implement a token bucket as a safety net.

2. How do you handle API structure changes in scraped data?

Validate scraped data against an expected schema. Monitor field count and types. Alert if structure changes significantly. Implement versioned parsers that can handle multiple API versions simultaneously.

3. How do you detect and skip duplicate scraped data?

Compute a content hash (MD5 of JSON-sorted data) and compare against previously seen hashes. Use idempotency keys from the source (API response IDs, timestamps). Store deduplication state in Redis for persistence across runs.

4. How do you handle scraping failures?

Retry transient failures (network, 429 rate limit, 503) with exponential backoff (30s, 60s, 120s). Skip persistent failures (401, 403, 404) and alert immediately. Log all failures with response headers for debugging.

Challenge

Build a cron-based scraping system: (1) scheduler: collect threat intel every 15 minutes, competitor prices every 6 hours, SSL certs every 24 hours, (2) rate limiter: max 10 req/min per source with token bucket, (3) deduplication: content hash-based dedup stored in Redis, (4) error handling: retry (429/503 with backoff), skip (4xx except 429), alert (5 consecutive failures), (5) validation: schema check per data type, field count monitoring, structure change detection, (6) storage: raw data in S3, deduped data in PostgreSQL, (7) monitoring: collection age, gap detection, success rate per source.

FAQ

Mini Project: Cron Scraping Platform

Build a cron-based data collection platform: (1) scraper registry: each source has URL, schedule (cron expression), rate limit, parser function, validation schema, (2) rate-limited HTTP client: token bucket (10 req/min per source), exponential backoff (30s-300s), jitter, (3) deduplication engine: content hash-based with Redis-backed seen set, (4) validation pipeline: schema validation (required fields, data types, ranges), structure change detection (field count, type changes), (5) storage writer: raw data to S3 (JSON lines), deduped data to PostgreSQL, (6) health monitor: scrapes per source, success rate, data age, collection gap, (7) alerting: on consecutive failures, stale data, structure changes.

What's Next

Now that you understand scheduled scraping with cron, explore email campaign scheduling, then learn about SSL certificate renewal.