Trust and Reliance in Consensus-Based Explanations from an Anti-Misinformation Agent

Reliance on AI explanations can create a false sense of consensus: The study found that when users rely heavily on an AI agent's explanations about misinformation, they may develop an "illusion of consensus"—believing multiple independent sources agree when they're actually drawing from the same underlying information. This is a critical concern for anti-misinformation tools.

Trust in AI doesn't necessarily prevent consensus illusions: Surprisingly, the research found that how much users trust an AI system had no significant effect on whether they fell prey to the consensus illusion. This suggests that user confidence in an AI tool alone is insufficient to prevent this cognitive bias.

How users behave with AI explanations matters more than their attitudes: The study distinguished between trust (what people believe about the system) and reliance (how they actually use it). Reliance—the behavioral component—was the key factor influencing the consensus illusion, indicating that the way explanations are presented and used is more important than building user confidence.

Explainable AI systems need user-centered design to combat misinformation effectively: The findings suggest that anti-misinformation AI tools should be carefully designed with attention to how explanations influence user behavior, not just whether users trust the system. This could involve redesigning how consensus-based explanations are presented to users.