Isolating LLM Lexical Bias: A Curation-Free Triangulated Metric for Preference-Stage Learning

Preference training introduces systematic word biases: Large language models develop overused words and phrases (like "delve" and "furthermore") during the preference-learning stage (e.g., reinforcement learning from human feedback), even when these patterns don't exist in the base model, representing a measurable misalignment with natural language.

A new automated detection method removes manual bias: The researchers developed the Triangulated Preference Shift score, which automatically identifies lexical changes caused by preference training without requiring manual curation, making it easier to detect and monitor these biases across different AI models.

Preference training may inadvertently create "prestige language": The analysis suggests that preference learning shifts models toward using more formal or prestigious language patterns, which could introduce subtle biases about what constitutes "correct" or "desirable" communication.

The problem spans multiple model families: Testing across six different model families demonstrates that lexical bias from preference training is a widespread issue, not isolated to individual models, suggesting this is a systemic concern in how modern AI systems are trained.

Better monitoring tools support trustworthy AI development: By providing an automated way to measure preference-training side effects, this metric can help AI developers identify and address unintended behavioral shifts, contributing to more transparent and aligned AI systems.