From XAI to Stories: A Factorial Study of LLM-Generated Explanation Quality

XAI methods provide minimal benefit for non-experts: The study found that traditional explainability techniques like SHAP and LIME only produced small improvements over no explanation at all, and primarily benefited expert audiences—raising questions about whether the complexity of converting these methods to natural language explanations is justified for general users.

The choice of AI model matters more than explanation technique: Across all tested factors, the specific Large Language Model used to generate explanations (DeepSeek-R1, GPT-4o, or Llama-3) had the largest impact on explanation quality—suggesting that organizations should prioritize selecting capable LLMs over perfecting explanation methods.

More complex explanations aren't always better: The study revealed a counterintuitive finding where simpler, more accurate forecasting models (like SARIMAX) actually produced lower-quality natural language explanations than more complex machine learning models, indicating that prediction accuracy alone doesn't guarantee understandable explanations.

Simple prompting strategies are as effective as complex ones at lower cost: Zero-shot prompting (asking the model to explain without examples) performed competitively with more sophisticated techniques like self-consistency, while requiring seven times less computational resources—suggesting organizations can save costs without sacrificing explanation quality.

Advanced prompting techniques can backfire: Chain-of-thought prompting, which asks models to "think through" explanations step-by-step, actually degraded explanation quality rather than improving it, cautioning against assuming that more elaborate prompting strategies automatically produce better results.