Documentation Testing — Complete Guide

In this tutorial, you will learn about Documentation Testing. We cover key concepts, practical examples, and best practices to help you master this topic.

Documentation testing validates accuracy by testing code examples, verifying instructions, checking for broken links, and ensuring the documentation matches the current software behavior.

What You'll Learn

You will learn how to test documentation automatically and manually, how to verify code examples, how to use doc testing frameworks, and how to make doc testing part of your CI pipeline.

Why It Matters

Outdated or incorrect documentation is worse than no documentation. Readers who follow instructions and get errors lose trust in both the docs and the product. Testing prevents this.

Real-World Use

DodaTech runs documentation testing as part of the CI pipeline. Code examples are extracted and executed, instructions are validated against actual behavior, and link checking catches broken references.

flowchart LR
  A[Documentation] --> B[Code Example Testing]
  A --> C[Instruction Validation]
  A --> D[Link Checking]
  A --> E[Spell Checking]
  B --> F{All Tests Pass?}
  C --> F
  D --> F
  E --> F
  F -->|Yes| G[Deploy]
  F -->|No| H[Fail CI]
  A:::current
  classDef current fill:#f90,color:#fff,stroke:#333,stroke-width:2px

Testing Code Examples

The most critical test: ensure code examples actually work.

# test_docs_examples.py
import subprocess
import pytest

def test_curl_example():
    """Verify the curl example in authentication docs works."""
    # Extract and run the curl command from the doc
    result = subprocess.run(
        ["curl", "-s", "-o", "/dev/null", "-w", "%{http_code}",
         "https://api.example.com/health"],
        capture_output=True, text=True
    )
    assert result.stdout.strip() == "200"

Using doctest for Python Documentation

def calculate_rate_limit(requests_per_minute):
    """
    Calculate the rate limit window size.

    Example:
        >>> calculate_rate_limit(60)
        1.0
        >>> calculate_rate_limit(120)
        0.5
    """
    return 60.0 / requests_per_minute

Run:

python -m doctest docs/examples.py

Expected output (if all tests pass): No output (exit code 0).

Command-Line Instruction Testing

Test that installation instructions work:

#!/bin/bash
# test_installation.sh
# Test the installation guide instructions

# Step 1: Download
curl -sL https://example.com/release.tar.gz -o /tmp/release.tar.gz
if [ $? -ne 0 ]; then
    echo "FAIL: Download step failed"
    exit 1
fi

# Step 2: Extract
tar -xzf /tmp/release.tar.gz -C /tmp/
if [ $? -ne 0 ]; then
    echo "FAIL: Extraction step failed"
    exit 1
fi

# Step 3: Run
/tmp/release/bin/app --version
if [ $? -ne 0 ]; then
    echo "FAIL: Application failed to start"
    exit 1
fi

echo "PASS: Installation instructions verified"

Automated Doc Testing with wrata

# .wrata.yaml
files:
  - path: content/**/*.md
    commands:
      - name: "Test code blocks"
        pattern: "```bash"
        run: |
          bash -c '{code}'

Extract and Test Code Snippets

import re
import subprocess

def test_markdown_code_blocks():
    """Extract and test all bash code blocks from docs."""
    with open("docs/installation.md") as f:
        content = f.read()

    # Extract bash code blocks
    blocks = re.findall(
        r"```bash\n(.*?)```", content, re.DOTALL
    )

    for i, block in enumerate(blocks):
        result = subprocess.run(
            ["bash", "-c", block],
            capture_output=True, text=True
        )
        assert result.returncode == 0, \
            f"Block {i+1} failed: {result.stderr}"

Common Mistakes

1. Testing Only Code, Not Instructions

Code examples may work, but instructions may skip steps or assume knowledge. Test both the code and the surrounding step-by-step guidance.

2. Using Outdated Test Data

Tests that pass against stale test data miss real issues. Refresh test data regularly.

3. Ignoring Environment Differences

Instructions that work on macOS may fail on Linux. Test on all supported platforms.

4. Not Testing Edge Cases

Documentation often covers the happy path. Test edge cases like empty responses, error codes, and Rate Limiting.

5. Manual Testing Without Documentation

If a human must test, document the test procedure. Otherwise, the test is not reproducible.

Practice Questions

1. Why is documentation testing important?

Incorrect documentation causes reader frustration, support tickets, and loss of trust. Testing catches issues before readers encounter them.

2. How do you test code examples in documentation?

Extract code blocks from Markdown and execute them in a test framework. Verify the output matches expected results.

3. What is the difference between doc testing and link checking?

Doc testing verifies content accuracy (code examples work, instructions are correct). Link checking verifies that references resolve.

4. How often should documentation tests run?

On every Pull Request for code examples. Comprehensive tests on every merge to main.

5. Challenge: Write a script that extracts all code blocks from a Markdown file and runs them, reporting which ones pass and fail. Include at least two programming languages.

FAQ

Mini Project

Create a test script that extracts all bash code blocks from a documentation page and runs them in a Docker container. Write a pytest test that validates Python code examples from documentation files.

What's Next

Testing ensures accuracy. Next, learn about Docs-as-Code Templates to standardize documentation structure. Then explore Multi-Version Docs.