Faithfulness Evaluation for Decoder-only LLM Attributions with Controlled Retained Information

Existing attribution evaluation methods are biased: Current popular metrics (Soft-NC and Soft-NS) for measuring whether LLM explanations are faithful can be misleading because methods that keep more words tend to get artificially higher scores, regardless of explanation quality.

A fairer comparison framework is needed: The researchers propose π-Soft-NC and π-Soft-NS, which evaluate attribution methods while controlling for the number of words retained, enabling more honest comparisons of which explanation methods actually work best.

No single "best" explanation method exists: Testing on real tasks with Llama and Mistral models showed that while Grad-ELLM performed well on some metrics, "there is no dominant method under π-Soft-NS," suggesting different explanation approaches may be better suited for different purposes.

Rigorous evaluation supports responsible AI development: By establishing clearer standards for measuring explanation quality, this work helps ensure that as LLMs are deployed in high-stakes applications, we can actually verify whether the reasons they provide for their outputs are trustworthy.

Explanation methods need task-specific evaluation: The research tested attribution methods on both classification and open-generation tasks, implying that organizations should evaluate explanation quality on the specific types of tasks their LLMs will perform.