Benchmarking Political Persuasion Risks Across Frontier Large Language Models

LLMs are more persuasive than traditional political advertising: Across two survey experiments with 19,145 participants, the study found that state-of-the-art large language models outperformed standard campaign advertisements at changing political views, indicating a meaningful escalation in persuasion risks compared to existing campaign practices.

Persuasiveness varies significantly between models: Claude models demonstrated the highest persuasiveness while Grok showed the lowest, revealing that different AI systems pose different levels of political influence risk—a finding that challenges assumptions that all frontier LLMs present equivalent threats.

Information-based prompts have unpredictable effects across models: Contrary to previous research suggesting that adding factual information increases persuasiveness, this study found the effect is model-dependent: information boosted Claude and Grok's persuasiveness but substantially reduced GPT's effectiveness, highlighting the complexity of predicting how different systems will behave.

A new framework for assessing persuasion risks is needed: The researchers developed an LLM-assisted analysis approach to identify underlying persuasive strategies, providing a data-driven method for comparing political influence risks across different models—a tool that could help policymakers and companies evaluate frontier AI systems more systematically.

Frontier model capabilities warrant ongoing monitoring: The study emphasizes that recent advances in LLM capabilities have created new persuasion risks that differ from those identified in earlier research, suggesting the need for continuous benchmarking as these systems continue to evolve.