Tuning for Trustworthiness -- Balancing Performance and Explanation Consistency in Neural Network Optimization

Explainability is typically ignored during AI model development: The paper reveals that current practice focuses almost exclusively on minimizing prediction errors during hyperparameter tuning, with explainability rarely considered—meaning models optimized the traditional way may produce unreliable explanations even if they perform well.

Different explanation methods often disagree with each other: The research introduces "XAI consistency" as a measure of how much different explanation techniques agree about which features matter. Low consistency suggests the model's explanations may not be trustworthy, even if predictions are accurate.

There's a trade-off zone between performance and trustworthiness: The study identifies three distinct regions in model design space—one with poor performance, one with good performance but unreliable explanations, and crucially, a "trade-off region" that achieves both competitive accuracy and consistent, interpretable explanations.

Explainability can be built into the model selection process: By incorporating XAI consistency directly into the optimization framework alongside performance metrics, developers can deliberately steer toward models that are both accurate and trustworthy, rather than treating explainability as an afterthought.

This approach provides practical tools for responsible AI development: The framework is implemented in the Sequential Parameter Optimization Toolbox (SPOT), making it actionable for practitioners who want to develop AI systems that are both performant and genuinely interpretable.