Prediction certification cannot replace explanation certification: a competence envelope for trustworthy AI under compound stress

Prediction accuracy alone is insufficient for trustworthy AI: The paper proves through a "separation theorem" that two AI models—one reliable and one compromised—can produce identical accuracy, calibration, and coverage metrics while behaving completely differently in real-world deployment, meaning current standard certifications miss critical failures.

Explanation access is essential for safety: Detecting when models fail in ways invisible to prediction metrics requires examining how models make decisions, not just what predictions they output. This means trustworthy AI systems need transparency into their decision-making mechanisms alongside performance measurements.

High confidence scores can mask problems: Models can maintain high accuracy and calibration (confidence in their predictions) while having fundamentally flawed reasoning, creating a false sense of security in high-stakes applications like medical diagnosis or building safety assessment.

The "competence envelope" framework combines two types of certification: The authors propose a practical approach that evaluates both prediction performance (accuracy, calibration) and explanation fidelity (how the model actually reasons), catching failure modes that either approach alone would miss.

Current AI certification practices are incomplete: Existing safeguards used to validate AI systems for consequential decisions rely too heavily on prediction-side metrics and leave dangerous blind spots, particularly under stressful or unusual conditions where models may fail in ways their accuracy scores don't reveal.