ExplainabilityTrustworthinessTransparencycs.AI
Knowledge-intensive Language Understanding for Explainable AI
Amit Sheth, Manas Gaur, Kaushik Roy, Keyur Faldu
August 2, 2021
IEEE Internet Computing
60 citations
1 influential
Key Takeaways
-
Trust requires understanding: The paper emphasizes that people can only trust AI systems when they understand how decisions are made and what factors influenced those decisions—a principle that applies across healthcare, finance, criminal justice, and other high-stakes domains.
-
Current explanations fall short of human decision-making: While AI systems can explain technical aspects like data biases and feature importance, they often lack "human-centered explanations" that mirror how domain experts actually reason through problems using established professional knowledge and guidelines.
-
Domain knowledge is essential for validation: The paper argues that validating whether an AI system's outputs are trustworthy requires incorporating explicit domain knowledge—the kind that human professionals understand and rely on—rather than relying solely on technical metrics.
-
Explainability is foundational to responsible AI: Beyond fairness and privacy concerns, explainability is positioned as a critical pillar for developing trustworthy AI systems, suggesting that opacity itself is a barrier to responsible deployment in sensitive applications.
Abstract
AI systems have seen significant adoption in various domains. At the same time, further adoption in some domains is hindered by inability to fully trust an AI system that it will not harm a human. Besides the concerns for fairness, privacy, transparency, and explainability are key to developing trusts in AI systems. As stated in describing trustworthy AI "Trust comes through understanding. How AI-led decisions are made and what determining factors were included are crucial to understand." The subarea of explaining AI systems has come to be known as XAI. Multiple aspects of an AI system can be explained; these include biases that the data might have, lack of data points in a particular region of the example space, fairness of gathering the data, feature importances, etc. However, besides these, it is critical to have human-centered explanations that are directly related to decision-making similar to how a domain expert makes decisions based on "domain knowledge," that also include well-established, peer-validated explicit guidelines. To understand and validate an AI system's outcomes (such as classification, recommendations, predictions), that lead to developing trust in the AI system, it is necessary to involve explicit domain knowledge that humans understand and use.