Source framing triggers systematic evaluation bias in Large Language Models

LLMs show strong agreement when evaluating text blindly, but this consensus collapses when source information is revealed. Across four major models tested on 4,800 statements (192,000 total assessments), inter- and intra-model agreement remained "remarkably high" in blind conditions, but broke down once source framing was introduced.

Attributing statements to Chinese sources systematically lowered evaluation scores across all models tested. The paper found that disclosing a Chinese origin for statements consistently reduced agreement scores, with Deepseek Reasoner showing particularly pronounced bias—raising concerns about geographic and nationality-based discrimination in AI evaluation systems.

Source framing effects undermine the neutrality and fairness of AI-mediated information systems. Since LLMs are increasingly used to evaluate content in real-world applications, these framing biases could distort how information is filtered, ranked, or assessed in systems that millions of people rely on for news and decision-making.

The same AI models cannot be trusted to evaluate information consistently when context about the source is available. This finding suggests that using LLMs as neutral arbiters of text quality or truthfulness is problematic, as their judgments appear vulnerable to the same social biases that affect human evaluators—potentially amplifying rather than mitigating bias in information systems.