User Agent Blocking — Block Bad Bots

In this tutorial, you'll learn about User Agent Blocking. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

User Agent Blocking lets you block or challenge requests based on the User-Agent HTTP header — a fast way to stop known bad bots, scraper tools, and outdated clients at Cloudflare's edge.

What You Will Learn

You will learn how to identify malicious User-Agent strings, create blocking rules through the dashboard and API, and combine this with WAF for layered bot mitigation.

Why It Matters

Bad bots account for over 25% of all web traffic. Blocking them at the edge saves bandwidth, reduces server load, and protects content from scraping without affecting legitimate users.

Real-World Use Case

A content site notices 40% of traffic comes from python-requests and curl with no session cookies. A User Agent Blocking rule challenges these requests, cutting server load by half and improving response times for real visitors.

How User Agent Blocking Works

When Cloudflare receives a request, it inspects the User-Agent header before any application logic runs. If the string matches a block or challenge rule, the request is terminated or challenged immediately.

flowchart LR
  A[Incoming Request] --> B{Parse User-Agent}
  B -->|Known Bot| C["Block / Challenge"]
  B -->|Legitimate UA| D[Pass to WAF]
  B -->|Empty / Missing| E[Challenge]
  C --> F[403 or JS Challenge]
  D --> G[Origin Server]
  E --> F

Creating a Rule via Dashboard

1. Go to Security > WAF > Tools in your Cloudflare dashboard.
2. Under User Agent Blocking, click Create rule.
3. Enter the User-Agent string or pattern (supports wildcards with *).
4. Choose action: Block (returns 403) or Challenge (JS challenge page).
5. Click Add to activate.

Common patterns to block:

API: Block a User-Agent Pattern

curl -X POST "https://api.cloudflare.com/client/v4/zones/ZONE_ID/firewall/access_rules/rules" \
  -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
  -H "Content-Type: application/json" \
  --data '{
    "mode": "block",
    "configuration": {
      "target": "ua",
      "value": "python-requests*"
    },
    "notes": "Block python-requests scraper"
  }'

Expected output:

{
  "result": {
    "id": "rule_id",
    "mode": "block",
    "notes": "Block python-requests scraper"
  },
  "success": true
}

Node.js: Check Blocked User-Agents

Use Node.js to fetch and monitor blocked Agent traffic from Cloudflare's analytics API:

const fetch = require("node-fetch");

const ZONE_ID = process.env.CLOUDFLARE_ZONE_ID;
const TOKEN = process.env.CLOUDFLARE_API_TOKEN;

async function getBlockedAgents() {
  const resp = await fetch(
    `https://api.cloudflare.com/client/v4/zones/${ZONE_ID}/analytics/dashboard`,
    {
      headers: {
        Authorization: `Bearer ${TOKEN}`,
        "Content-Type": "application/json",
      },
    }
  );
  const data = await resp.json();
  console.log("Blocked requests:", data.result.totals.blocked);
}

getBlockedAgents();

Expected output:

Blocked requests: 15423

Go: Validate User-Agent Before Sending

When building clients that should not be blocked, set a proper User-Agent:

package main

import (
  "fmt"
  "net/http"
  "os"
)

func main() {
  client := &http.Client{}
  req, _ := http.NewRequest("GET", "https://example.com", nil)
  req.Header.Set("User-Agent", "MyApp/1.0 (legitimate bot; contact@example.com)")
  resp, err := client.Do(req)
  if err != nil {
    fmt.Println("Error:", err)
    os.Exit(1)
  }
  defer resp.Body.Close()
  fmt.Println("Status:", resp.StatusCode)
}

Expected output:

Status: 200

Common Mistakes

Practice Questions

1. What action should you use instead of Block if you want to give suspicious visitors a chance to prove they are human?
2. How do you use wildcards in User Agent Blocking patterns?
3. Why is User Agent Blocking alone insufficient against advanced bots?

Challenge

Create a script that reads a daily feed of known bad User-Agent strings and syncs them to Cloudflare using the API. Include error handling for duplicate entries.

Real-World Task

Your forum is being spammed by a bot using a fake User-Agent that mimics Chrome. Explain how you would identify the bot pattern using Cloudflare analytics and create a targeted blocking rule that does not affect real Chrome users.

FAQ

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro — security-first tools for the modern web.