Evaluating Explainable Methods for Predictive Process Analytics: A Functionally-Grounded Approach

Popular explanation tools may not work equally well across all AI applications: The paper tests LIME and SHAP—two widely-used methods for explaining AI decisions—on business process prediction models and finds that their effectiveness varies. This suggests organizations cannot assume these tools will provide reliable explanations in every context where they're applied.

"Black box" AI predictions in business processes need better transparency solutions: The research identifies a gap between the accuracy of advanced machine learning models (like XGBoost) used to predict business outcomes and our ability to explain why those predictions are made, highlighting a real-world trustworthiness problem in enterprise AI systems.

Current evaluation methods for AI explanations may be insufficient: The authors propose new "functionally-grounded" metrics specifically designed for process analytics, suggesting that existing ways of measuring whether AI explanations are good enough may not capture what actually matters for business users who need to understand and act on predictions.

Testing AI explanations on real-world data is essential: By evaluating explanation methods across three real open-source event logs rather than just synthetic data, the research demonstrates that explanation quality can vary significantly depending on the actual business context—an important consideration for organizations deploying these tools.

Human trust in AI requires more than just adding explanations: The paper emphasizes "human user-oriented evaluation," suggesting that simply attaching an explanation tool to an AI system doesn't automatically make it trustworthy; organizations need to verify that explanations actually help users understand and appropriately rely on AI predictions.