Illocutionary Explanation Planning for Source-Faithful Explanations in Retrieval-Augmented Language Models

LLMs produce persuasive but unverifiable explanations: The study found that non-retrieval-augmented language models have 0% median source adherence, meaning their explanations cannot be traced back to authoritative sources—a critical problem when users need to verify whether claims are actually supported by evidence.

Even retrieval-augmented systems struggle with faithfulness: Standard retrieval-augmented generation (RAG) systems, which are designed to ground responses in source material, still only achieved 22-40% median source adherence across different models, indicating that simply adding retrieval doesn't automatically solve the problem of trustworthy explanations.

A new prompting technique significantly improves source fidelity: The researchers' "chain-of-illocution" (CoI) method—which breaks down queries into implicit explanatory questions to guide retrieval—achieved up to 63% improvements in source adherence, suggesting that how we prompt LLMs matters substantially for responsible AI deployment.

Better source traceability doesn't reduce user satisfaction: A user study with 165 participants showed that explanations with improved source adherence did not harm user satisfaction, suggesting there is no inherent trade-off between making AI systems more verifiable and maintaining good user experience.

The problem remains unsolved at scale: Despite improvements, absolute source adherence remained "moderate" even with the new method, and gains were weak or non-significant for some models, indicating that source-faithful explanations in LLMs remain an ongoing challenge requiring further research.