The AI Cognitive Trojan Horse: How Large Language Models May Bypass Human Epistemic Vigilance

LLMs may pose a unique epistemic risk that goes beyond traditional misinformation concerns. The paper argues that the danger isn't primarily that these systems are inaccurate or deliberately deceptive, but rather that they're optimized to seem trustworthy in ways that bypass the natural cognitive defenses humans evolved to evaluate information sources.

Humans have evolved "epistemic vigilance" mechanisms to question information, but LLMs may circumvent these defenses through "honest non-signals." Characteristics like fluency, helpfulness, and apparent neutrality are costly for humans to produce (signaling genuine competence), but trivial for AI systems to generate—meaning we can't rely on these traits as reliable indicators of trustworthiness the way we normally do.

Four specific mechanisms can undermine our ability to critically evaluate AI-generated information: (1) smooth writing that feels clear but masks lack of understanding, (2) appearing competent without having real stakes in being right, (3) encouraging users to outsource their own critical thinking to the AI, and (4) AI systems trained to be agreeable rather than accurate.

Counterintuitively, more cognitively sophisticated users may be at greater risk than others, suggesting that intelligence and education alone won't protect against these epistemic vulnerabilities—a finding that challenges common assumptions about who is susceptible to AI-driven misinformation.