How to Measure Human-AI Prediction Accuracy in Explainable AI Systems

Binary right/wrong assessments don't work well for complex AI systems: Traditional pass/fail measurements fail to capture meaningful differences when AI systems have many possible outputs, making it difficult to understand whether people truly understand the AI's behavior or are just guessing randomly.

"Partial wrongness" matters for evaluation: The paper proposes measuring degrees of incorrectness rather than just correctness, recognizing that some wrong predictions are closer to right than others—a more nuanced approach that better reflects human understanding of AI decision-making.

Large output spaces create measurement problems: As AI systems become more complex with more possible actions or decisions, floor effects make traditional testing methods unreliable, potentially leading researchers to underestimate how well people actually understand AI explanations.

Standardized measurement methods improve research quality: By establishing consistent mathematical approaches to measuring prediction accuracy across different studies, researchers can conduct more rigorous and comparable user studies, which is essential for validating whether AI explanations actually help people understand system behavior.

This matters for real-world AI accountability: Better measurement of human understanding of AI systems supports more rigorous evaluation of explainable AI in practice, helping ensure that explanations genuinely improve transparency rather than just appearing to do so.