How to explain it to data scientists? A mixed-methods user study about explainable AI, using mental models for explanations

Data scientists need explanations tailored to their specific role: Unlike general users, data scientists require explanation content drawn from three distinct domains—the application domain, system domain, and AI domain—reflecting their unique responsibility for training, testing, deploying, and maintaining ML models.

Complex explanations work better when organized strategically: The study found that explanation content should be organized sequentially and/or hierarchically rather than presented all at once, helping data scientists navigate technical information more effectively.

Explanations should tell a "causal story": Rather than listing isolated facts, explanations are most useful when organized as a coherent narrative that shows cause-and-effect relationships, helping data scientists understand not just what a model does but why.

Different explanation goals require different content types: The research identified that different explanation intents (such as understanding reasons, making comparisons, assessing accuracy, making predictions, or building trust) each require specific types of explanation content, suggesting one-size-fits-all explanations are insufficient.

Standardized explanation questions improve completeness: Using a structured set of explanation questions helps ensure that all relevant explanation needs are addressed, reducing gaps in understanding and supporting more thorough model evaluation and maintenance.