Creation of User Friendly Datasets: Insights from a Case Study concerning Explanations of Loan Denials

Data quality matters as much as algorithm design: The paper reveals that most explainability research focuses on making algorithms better at explaining decisions, but the underlying data often contains features that ordinary users cannot understand regardless of how well the AI explains them. This suggests that responsible AI requires attention to data creation, not just algorithm development.

Existing datasets fail real users: The researchers found that "existing datasets seldom contain features that end users consider as acceptable in understanding a model's decision." This gap between what AI engineers think is explainable and what actual users need highlights a critical blind spot in current AI development practices.

Context shapes what explanations should look like: Understanding who is seeking the explanation and why they need it is essential for creating truly user-friendly AI systems. The paper demonstrates this by creating explanations for loan denials that serve different purposes—some to educate applicants and others to help them take concrete steps toward future approval—rather than one-size-fits-all explanations.

User-centered design should start at the data stage: Rather than treating explainability as a problem to solve at the end of AI development, this research shows that involving end-users in dataset creation from the beginning produces more meaningful and actionable explanations for the people actually affected by AI decisions.