NeuroFaith: Evaluating LLM Self-Explanation Faithfulness via Internal Representation Alignment

LLMs can generate convincing but misleading explanations: The paper demonstrates that large language models produce plausible-sounding explanations for their answers that may not actually reflect how the model reached its conclusion, raising concerns about transparency in AI systems.

Current evaluation methods are insufficient: Existing approaches to checking whether AI explanations are truthful rely on behavioral tests or computational analysis, but don't examine whether the concepts mentioned in explanations actually matter to the model's decision-making process.

NeuroFaith offers a new verification approach: The proposed framework tests faithfulness by identifying key concepts in explanations and mechanistically checking whether those concepts genuinely influence the model's predictions, providing a more rigorous standard for trustworthiness.

The method works across different task types: The research validates NeuroFaith on both multi-step reasoning tasks and classification problems, suggesting the approach has broad applicability rather than being limited to specific use cases.

Unfaithful explanations can be detected and improved: The paper shows it's possible to identify unreliable explanations from the model's internal representations and then steer the model toward more faithful explanations, offering a practical path toward more trustworthy AI systems.