Evaluating Trustworthiness of AI-Enabled Decision Support Systems: Validation of the Multisource AI Scorecard Table (MAST)

MAST provides a practical checklist for building trustworthy AI systems: The study validates that the Multisource AI Scorecard Table (MAST)—a tool based on analytic tradecraft standards—can effectively guide the design of AI systems that people actually trust. When 177 subject matter experts tested AI systems designed with high-MAST ratings versus low-MAST ratings, trust perceptions were "highly correlated" with MAST scores, suggesting the tool works as intended.

Transparency and inspectability matter for trust, even with complex AI: One of the paper's key challenges is that many AI systems use "complex deep learning algorithms that drive DSS performance and preclude complete manual inspection." MAST addresses this by focusing on system components and capabilities that can be evaluated and explained, helping organizations build trust despite the "black box" nature of underlying algorithms.

Trust in AI can be systematically measured and improved: Rather than treating trust as vague or unmeasurable, this research demonstrates that specific design choices—captured in MAST criteria—directly influence whether people trust AI decision support systems. This suggests organizations have concrete levers to pull when deploying AI in high-stakes domains like security screening or investigative reporting.

Real-world testing with experts validates theoretical frameworks: The study didn't just survey people about trust; it created two interactive test environments (identity verification and text summarization tasks) where experts actually used the systems. This hands-on validation approach strengthens confidence that MAST reflects what genuinely builds trust in practice.