ExplainabilityTransparencyTrustworthinesscs.AI
From Data to Commonsense Reasoning: The Use of Large Language Models for Explainable AI
Stefanie Krause, Frieder Stolzenburg
July 4, 2024
arXiv.org
6 citations
Key Takeaways
-
LLMs can outperform humans at commonsense reasoning: Llama 3 achieved an average accuracy of 90% across eleven datasets, performing approximately 21% better than humans on average, suggesting these models have developed sophisticated reasoning capabilities that exceed typical human performance on question-answering tasks.
-
Explainability varies significantly between models: While GPT-3.5 received positive ratings for explanation quality (66% of survey participants rated explanations as "good" or "excellent"), the paper indicates that explainability performance differs across the three tested models (GPT-3.5, Gemma, and Llama 3), highlighting that model choice matters for transparency.
-
Current LLMs show promise for responsible AI applications: The ability of large language models to provide both accurate answers and human-like explanations addresses a key requirement for responsible AI deployment, particularly in high-stakes domains like question-answering systems where users need to understand the reasoning behind decisions.
-
Performance varies widely across different tasks: GPT-3.5's accuracy ranged from 56% to 93% depending on the specific QA benchmark, indicating that LLM reliability is task-dependent and cannot be assumed to be uniformly strong across all applications.
Abstract
Commonsense reasoning is a difficult task for a computer, but a critical skill for an artificial intelligence (AI). It can enhance the explainability of AI models by enabling them to provide intuitive and human-like explanations for their decisions. This is necessary in many areas especially in question answering (QA), which is one of the most important tasks of natural language processing (NLP). Over time, a multitude of methods have emerged for solving commonsense reasoning problems such as knowledge-based approaches using formal logic or linguistic analysis. In this paper, we investigate the effectiveness of large language models (LLMs) on different QA tasks with a focus on their abilities in reasoning and explainability. We study three LLMs: GPT-3.5, Gemma and Llama 3. We further evaluate the LLM results by means of a questionnaire. We demonstrate the ability of LLMs to reason with commonsense as the models outperform humans on different datasets. While GPT-3.5's accuracy ranges from 56% to 93% on various QA benchmarks, Llama 3 achieved a mean accuracy of 90% on all eleven datasets. Thereby Llama 3 is outperforming humans on all datasets with an average 21% higher accuracy over ten datasets. Furthermore, we can appraise that, in the sense of explainable artificial intelligence (XAI), GPT-3.5 provides good explanations for its decisions. Our questionnaire revealed that 66% of participants rated GPT-3.5's explanations as either "good" or "excellent". Taken together, these findings enrich our understanding of current LLMs and pave the way for future investigations of reasoning and explainability.