Does a model's stated reason for rejecting a candidate do any work?

Language models often cite reasons for rejecting candidates that don't actually influence their decisions. When researchers added the specific facts that models claimed were missing from a candidate's profile, the models' choices only sometimes changed—and the effect was weaker than when irrelevant sentences were added, suggesting the stated reasons may be post-hoc rationalizations rather than genuine decision factors.

Models' explanations can be misleading about what drives their choices. The study found that identical irrelevant sentences actually moved models' decisions more when placed at the candidate the model had criticized than when placed elsewhere, indicating that factors like sentence placement and fluency matter more than the logical content of the stated reason.

Current evaluation methods may not catch when AI systems confabulate explanations. Because language models can produce fluent, specific-sounding reasons that sound plausible but don't reflect their actual decision-making process, relying on their stated justifications without independent verification could lead to false confidence in AI transparency and accountability.

Testing AI reasoning requires going beyond accepting stated explanations. The researchers demonstrated that explanations need to be empirically validated—by manipulating the factors the model claims matter and observing whether choices actually change—rather than simply trusting that models accurately report their own reasoning processes.