The Impact of Stochasticity on the Rashomon Effect in Machine Learning

Multiple sources of randomness in AI training create different types of uncertainty: The paper identifies three distinct sources of stochasticity (weight initialization, data ordering, and dropout) that don't all contribute equally to the Rashomon effect, meaning that different aspects of AI model randomness need to be understood and managed separately rather than as a single problem.

Having many mathematically equivalent models doesn't guarantee they'll make the same predictions: The research reveals that even when neural networks achieve similar performance levels, they may disagree on specific predictions—a critical finding for high-stakes applications where consistency and reliability matter, such as healthcare or criminal justice.

AI explanations can be misleading about true model agreement: The study found that explanation maps (tools used to show why an AI made a decision) can appear more consistent across different models than the actual predictions are, suggesting that current explainability methods may give false confidence about model reliability.

We need multiple measurements to assess AI model stability: Rather than relying on a single metric, the paper demonstrates that responsible AI evaluation requires examining three complementary dimensions—solution diversity, prediction consistency, and explanation agreement—to get a complete picture of whether different trained models will behave similarly in practice.