PoliticsBench: Benchmarking Political Values in Large Language Models with Multi-Turn Roleplay

LLMs show stronger political values in roleplay scenarios than direct questions: When eight prominent LLMs were asked about political values through multi-turn roleplay scenarios rather than straightforward political questions, they expressed significantly broader and more strongly held value positions—activating approximately 0.75 additional value dimensions out of 10 total dimensions.

Models become increasingly committed to political stances during conversations: As interactions progressed from initial responses to final decisions, LLMs increased their commitment to particular stances by approximately 1.4 points on a 5-point scale, suggesting that extended dialogue can amplify political positioning rather than maintain neutrality.

Current political bias benchmarks are too simplistic: Existing methods for measuring LLM political bias focus mainly on demographic stereotypes and use coarse-level assessments, missing the nuanced values that actually drive sociopolitical reasoning and decision-making in real-world applications.

Interactive context matters more than static prompts for revealing bias: The paper demonstrates that evaluating political behavior requires moving beyond single-prompt assessments to longer interactive settings, as scenario-based roleplay elicits more authentic value expression than isolated questions.

Robustness concerns emerge in extended interactions: While inter-judge agreement remained stable, responses became less robust to scenario paraphrasing as conversations progressed, raising questions about the consistency and reliability of LLM political outputs in real-world use.