Mind the Gap: How Elicitation Protocols Shape the Stated-Revealed Preference Gap in Language Models

The stated-revealed preference gap in AI systems is largely an artifact of how we measure it. The paper finds that when researchers allow language models to express neutrality or abstain from answering during preference elicitation, the correlation between what models say they value and what they actually do improves substantially—suggesting previous findings of a "gap" may have been partly measurement error rather than genuine inconsistency.

Forcing binary choices creates misleading results about AI alignment. The study shows that standard yes/no prompting "entangles genuine preferences with artifacts of the elicitation protocol," meaning current evaluation methods may not accurately reflect what language models actually prefer or how they will behave in real-world applications.

Allowing too much flexibility in responses can make the problem worse. While permitting neutrality during stated preferences helps, allowing models to abstain from choices entirely during revealed preference tasks drives correlation to near-zero or negative values due to high rates of non-commitment—suggesting researchers need to find a middle ground between forced choices and complete flexibility.

Steering models with their stated values doesn't reliably improve behavior. The paper tested whether reminding models of their stated preferences through system prompts would make their actual choices more consistent with those values, but found this approach "does not reliably improve" alignment on ethical dilemmas, raising questions about the effectiveness of this common alignment technique.

Measuring AI values requires rethinking methodology. The research demonstrates that preference elicitation is more complex than current practices assume and that evaluating whether AI systems are truly aligned with human values requires methods that account for genuine uncertainty and indeterminate preferences rather than forcing artificial certainty.