Let's start with the uncomfortable truth: even the best automated accessibility testing tool will miss roughly 43% of WCAG 2.1 AA success criteria. This isn't a product limitation — it's a fundamental constraint of what machines can evaluate without human judgment.
If you're relying solely on automated scans to declare your site accessible, you have significant blind spots. Here's what automation misses, why it matters, and how to build a testing strategy that actually provides confidence.
What Automated Testing Does Well
First, let's give automation its due. Automated tools excel at catching:
Structural Issues
- Missing alt text on images
- Heading hierarchy violations (skipping levels)
- Missing form labels and associations
- Empty links and buttons
- Missing page titles and language declarations
- Duplicate IDs
Visual Presentation
- Color contrast violations (text against background)
- Viewport meta restrictions (preventing zoom)
- Text size in fixed units
ARIA and Semantics
- Invalid ARIA attributes
- Missing required ARIA properties
- Roles without accessible names
- Landmark region issues
Interactive Elements
- Missing focus indicators (detectable via CSS analysis)
- Keyboard traps (some, via automated tab navigation)
- Autocomplete attribute presence
These checks are binary — either the alt attribute exists or it doesn't. Either the contrast ratio meets 4.5:1 or it doesn't. Machines handle binary questions perfectly.
What Automated Testing Cannot Assess
The criteria that automation misses require understanding context, meaning, and user experience — things machines don't have access to.
Image Quality, Not Just Presence
Automation verifies that an alt attribute exists. It can't tell you whether the alt text is meaningful:
alt="image"passes automated checks but fails the purpose of 1.1.1alt="Photo of building"passes but may miss critical context (which building? why is it relevant?)alt="City Hall at 123 Main Street, the location for Tuesday's public hearing"conveys actual meaning
Evaluating alt text quality requires understanding what information the image conveys in its specific context.
Logical Reading Order
A page might render visually in a sensible order while the DOM order (what screen readers follow) is completely different due to CSS flexbox/grid reordering. Automation can detect reordering but can't judge whether the resulting order is logical.
Meaningful Link Text
<a href="...">Click here</a> is programmatically valid but useless for screen reader users who navigate by link list. Automation can flag generic text patterns, but judging whether link text is descriptive enough requires understanding the destination.
Consistent Identification
WCAG 2.1 criterion 3.2.4 requires that components with the same function are identified consistently across pages. A search function called "Search" on one page and "Find" on another is inconsistent — but detecting this requires cross-page semantic comparison that's beyond current automation.
Error Identification and Suggestions
Automation can check that error messages exist, but not whether they're helpful:
- "Error" — technically present, completely unhelpful
- "Invalid input in field 3" — slightly better, still unclear
- "Phone number must include area code, e.g., (555) 123-4567" — actually useful
Cognitive Accessibility
Several criteria relate to cognitive load and understanding:
- Is the language clear and appropriate for the audience?
- Are instructions provided for complex interactions?
- Can users easily reverse or confirm consequential actions?
These require human judgment about the target audience and interaction complexity.
Motion and Timing
While automation can detect some timing issues, it can't assess:
- Whether auto-playing content can be paused effectively
- Whether timing limits are truly essential or arbitrary
- Whether motion in animations exceeds the "three flashes" threshold in all states
The Coverage Gap in Numbers
Research from multiple sources (including Deque's analysis and the W3C's ACT Rules community) consistently shows:
| Category | Automatable | Requires Manual Testing |
|---|---|---|
| Level A criteria (30) | ~18 fully, ~6 partially | ~6 fully manual |
| Level AA criteria (20) | ~10 fully, ~5 partially | ~5 fully manual |
| Total | ~57% fully automatable | ~25% manual-only, ~18% partial |
"Partially automatable" means the tool can flag potential issues but a human must verify. For example, automation can flag that an image has alt text of only one word — but a human must decide if that single word is actually sufficient for that image.
Building a Complete Testing Strategy
The answer isn't "don't use automated testing" — it's "use automated testing as one layer of a multi-layered approach."
Layer 1: Automated Scanning (Continuous)
Run automated scans continuously on every page. This catches the 57% efficiently and catches regressions immediately. HelloCivic Scanner handles this layer, running on schedule and on-demand.
Frequency: Daily or weekly for active sites Scope: Full site crawl Ownership: Automated, reviewed by accessibility lead
Layer 2: Semi-Automated Review (Monthly)
For the "partially automatable" criteria, set up workflows where tools flag potential issues and humans verify:
- Review auto-generated alt text suggestions
- Verify link text in context
- Check color contrast in complex gradients or overlapping elements
- Evaluate ARIA usage correctness (not just validity)
Frequency: Monthly or after major content changes Scope: High-traffic pages and new content Ownership: Content team with accessibility training
Layer 3: Manual Expert Testing (Quarterly)
For the fully manual criteria, trained testers evaluate the experience:
- Navigate the entire site with keyboard only
- Use screen readers (NVDA, JAWS, VoiceOver) through key user flows
- Assess reading order in complex layouts
- Evaluate cognitive accessibility of forms and processes
- Test with browser zoom at 200% and 400%
Frequency: Quarterly or before major launches Scope: Key user journeys (5–10 critical paths) Ownership: Accessibility specialist or external auditor
Layer 4: User Testing (Semi-Annually)
Nothing replaces actual users with disabilities navigating your site and providing feedback:
- Recruit participants with various disabilities
- Observe them completing real tasks on your site
- Document barriers, frustrations, and workarounds
- Prioritize fixes based on real user impact
Frequency: Semi-annually or annually Scope: Top 5 resident tasks Ownership: UX team or external research firm
The Integration Point
HelloCivic Scanner covers Layer 1 comprehensively and facilitates Layer 2 through its manual checks workflow. The platform stores all findings — automated and manual — in one place, so your VPAT and compliance reporting reflect your actual testing breadth.
Automated testing is essential. It's just not sufficient. Build the full stack, and you'll have genuine confidence in your accessibility posture — not just a green checkmark from a tool that's only seeing half the picture.