Key Takeaways on Model Explainability

• Explainability techniques may not be reliable across different scenarios: The paper found "inconsistency in FI [feature importance] identification between the static and dynamic cases," meaning that the factors identified as important for predictions can change depending on the situation—a critical concern when these models make real decisions about people's credit.

• Current explainability methods need improvement before deployment: The authors identify gaps in "the state of the art in model explainability," suggesting that existing techniques for understanding AI predictions are not yet mature enough to fully trust in high-stakes applications like lending decisions.

• What-if testing reveals hidden model behavior: By testing how models respond to hypothetical changes in loan applicant characteristics (both continuous factors like income and categorical factors like employment type), the researchers uncovered inconsistencies that static analysis alone would have missed—highlighting the importance of stress-testing AI systems before use.

• Feature importance alone is insufficient for transparency: The paper demonstrates that simply identifying which factors a model considers "important" doesn't guarantee you understand how the model actually works in practice, suggesting that responsible AI requires multiple layers of verification.

• Transparency in lending decisions requires ongoing scrutiny: Given the application to credit decisions, these findings underscore that organizations using AI for loans must continuously validate that their models' explanations remain consistent and trustworthy.