Exploring Trust Calibration in XAI - The Impact of Exposing Model Limitations to Lay Users

Telling users about AI limitations helps them judge performance more accurately: The study found that explicitly disclosing model limitations was the only reliable way to improve how well users' trust in an AI system matched its actual performance on individual cases, suggesting that transparency about what the system can and cannot do matters for responsible deployment.

General warnings about AI capabilities have limited effect: Providing users with example-based information about the AI system upfront (before they used it) did not significantly improve their ability to calibrate trust, indicating that one-time onboarding explanations may be less effective than ongoing, case-specific limitation communication.

What users actually experience matters more than how you frame it: The researchers found that the actual quality of predictions users encountered explained substantially more variance in trust calibration than the experimental conditions designed to shape expectations, suggesting that real performance is a stronger influence on user judgment than framing or instructions.

Users struggle to distinguish between related concepts: Participants had difficulty differentiating between perceived trust, trustworthiness, and accuracy when evaluating individual cases, which has implications for how XAI systems should communicate about model performance to avoid confusion.

Short-term experience alone doesn't improve judgment: Simply using the AI system for 15 cases did not lead to progressive improvement in how well users calibrated their trust to actual performance, suggesting that passive experience may not be sufficient for users to learn appropriate reliance on AI systems.