What Does Explainable AI Mean in Practice? Evaluative Requirements from a Longitudinal Clinical Case Study

Clinicians don't need to understand how AI works—they need to verify it works. The study found that doctors trusted the cerebral palsy prediction system not because they understood its internal mechanics, but because it let them compare the AI's predictions against their own clinical assessments, challenging the common assumption that detailed technical explanations build trust.

Simple visualizations outperform complex explanations. A straightforward prediction graph proved more effective than complicated technical documentation at helping clinicians make decisions, suggesting that effective AI explanations should match how experts already work rather than forcing them to learn new frameworks.

Trust in AI requires "Evaluative Requirements"—the ability to scrutinize outputs. The researchers introduced this concept to describe system features that let users critically examine AI predictions and compare them to their own expertise, positioning evaluation capability as essential rather than optional for trustworthy AI in high-stakes domains like healthcare.

Building trustworthy AI requires ongoing collaboration with actual users. Over 18 months of iterative prototyping and observation with both developers and hospital clinicians revealed what explanations actually worked in practice—findings that likely wouldn't have emerged from theoretical design alone, suggesting this collaborative approach should be standard practice.

One-size-fits-all explanations won't work across different domains. The study demonstrates that explainability requirements are context-specific and best discovered through direct engagement with expert users in their actual work environments, not through generic AI guidelines.