Evaluation Methods — Testing and Documenting VPAT Conformance

In this tutorial, you will learn about Evaluation Methods. We cover key concepts, practical examples, and best practices to help you master this topic.

VPAT evaluation methods include automated testing tools, manual keyboard testing, screen reader testing with NVDA or JAWS, manual code review, and user testing with people with disabilities to assign accurate conformance ratings.

What You'll Learn

You will learn the five main evaluation methods used to fill out a VPAT, how to choose the right method for each criterion, and how to document your testing process for credibility.

Why It Matters

Buyers trust VPATs that are backed by real testing. A VPAT filled out without evaluation is guesswork. Documented evaluation methods prove your conformance claims are reliable.

Real-World Use

DodaTech uses a three-method evaluation process for each VPAT update: automated scanning with axe DevTools, manual keyboard testing, and screen reader testing with NVDA. Each method targets different criteria.

flowchart TD
  A[Evaluation Methods] --> B[Automated Testing]
  A --> C[Keyboard Testing]
  A --> D[Screen Reader Testing]
  A --> E[Code Review]
  A --> F[User Testing]
  B --> G[axe, WAVE, Lighthouse]
  C --> H[Tab through all interactions]
  D --> I[NVDA, JAWS, VoiceOver]
  E --> J[Review HTML, ARIA, CSS]
  F --> K[Real users with disabilities]

Automated Testing

Automated tools scan pages and flag violations against WCAG success criteria. They catch about 30 percent of issues. Tools like axe DevTools, WAVE, and Google Lighthouse integrate into your development workflow.

// Run axe-core programmatically for VPAT evaluation
const AxeBuilder = require('@axe-core/webdriverjs');
const WebDriver = require('selenium-webdriver');

async function evaluatePage(url) {
  const driver = new WebDriver.Builder()
    .forBrowser('chrome')
    .build();

  try {
    await driver.get(url);
    const results = await new AxeBuilder(driver).analyze();

    const violations = results.violations.map(v => ({
      id: v.id,
      impact: v.impact,
      description: v.description,
      criteria: v.tags.filter(t => t.startsWith('wcag'))
    }));

    console.log(`Found ${violations.length} violations on ${url}`);
    return violations;
  } finally {
    await driver.quit();
  }
}

evaluatePage('https://example.com').then(console.log);

Expected output:

Found 4 violations on https://example.com
[
  { id: 'color-contrast', impact: 'serious', description: 'Elements must have sufficient color contrast', criteria: ['wcag21aa', 'wcag143'] }
]

Manual Keyboard Testing

Keyboard testing verifies every interactive element is reachable and operable with Tab, Enter, Escape, and arrow keys. No mouse allowed.

<!-- Test keyboard accessibility of a custom select -->
<div role="listbox" tabindex="0" aria-label="Sort products">
  <div role="option" aria-selected="true" tabindex="-1">Newest</div>
  <div role="option" aria-selected="false" tabindex="-1">Price: Low to High</div>
  <div role="option" aria-selected="false" tabindex="-1">Price: High to Low</div>
</div>

Expected behavior: Tab focuses the listbox. Arrow keys cycle through options. Enter selects the highlighted option. Escape closes any open dropdown.

Screen Reader Testing

Test with NVDA (free, Windows) or VoiceOver (built into macOS). Navigate using only screen reader commands. Verify that all content and functionality are announced correctly.

Documenting Methods in VPAT

Include a section in your VPAT describing which evaluation methods were used. This adds credibility and helps buyers understand the thoroughness of your testing.

<div class="evaluation-methods">
  <h3>Evaluation Methods Used</h3>
  <ul>
    <li>Automated: axe DevTools v4.8, WAVE Browser Extension</li>
    <li>Manual keyboard testing: Chrome, Firefox, Safari</li>
    <li>Screen reader: NVDA 2024, VoiceOver (macOS 14)</li>
    <li>Manual code review: HTML and ARIA audit</li>
  </ul>
  <p>Evaluation date: <time datetime="2026-06-28">June 28, 2026</time></p>
</div>

Common Mistakes

1. Relying Only on Automated Tools

Automated tools catch about 30 percent of issues. They miss context-dependent criteria like meaningful alt text.

2. Not Documenting the Evaluation Process

Without documentation, buyers cannot assess the reliability of your VPAT ratings.

3. Testing Only One Browser

Each browser and screen reader combination behaves differently. Test with at least two.

4. Skipping Keyboard Testing

Many criteria require keyboard testing. Automated tools cannot verify keyboard operability.

5. Using Outdated Testing Tools

Tools update their rule sets. Using an outdated version may produce incorrect results.

6. Not Testing on Mobile

WCAG applies to mobile too. Test on iOS and Android if your product has mobile interfaces.

7. Confusing Automated Pass with Full Conformance

A clean automated report does not mean the product is accessible. Manual testing is still required.

Practice Questions

1. What percentage of WCAG issues do automated tools typically catch?

About 30 percent. Manual methods are required for the remaining 70 percent.

2. Which screen readers should you test with?

NVDA (Windows), VoiceOver (macOS), and optionally JAWS (Windows) for enterprise coverage.

3. Why is keyboard testing essential for VPAT evaluation?

Many WCAG criteria require keyboard operability, which automated tools cannot verify.

4. What should you include in the evaluation methods section of a VPAT?

Tools used, versions, testing dates, browsers, and screen readers.

5. Challenge: Evaluate a page using all three methods: automated scan, keyboard testing, and screen reader testing. Compare the findings.

FAQ

Mini Project

Choose a web page from your product. Run axe DevTools automated scan, keyboard test all interactive elements, and test with NVDA screen reader. Document findings for each method.

What's Next

Learn how to write effective Remarks and Explanations in the criteria table. Then understand Legal Disclaimers in VPATs.