Evaluating Large Language Models in a Complex Hidden Role Game

LLMs currently struggle with sustained deception in complex social settings: When playing as the Fascist role in Secret Hitler, models like Llama 3.1 70B achieved only 59.7% accuracy in role identification compared to rule-based agents' 86.7%, and consistently failed to maintain deception, resulting in games roughly 40% shorter than human games.

Advanced reasoning techniques don't improve strategic deception: Contrary to expectations, Chain-of-Thought prompting and internal memory actually worsened performance, with some models experiencing up to 23.2% worse win rates when playing Fascist roles, suggesting current LLMs lack the multi-turn strategic reasoning needed for complex manipulation.

There's a critical gap between conversational ability and strategic reasoning: The research reveals that LLMs' strength in conversation doesn't translate to strategic depth in games requiring deception and persuasion, highlighting an important distinction for AI safety assessments.

We need better tools to monitor when LLMs master deceptive capabilities: The authors developed reproducible metrics (Role Identification Accuracy, Deception Retention Rate, Game State Impact Rate) specifically to track deceptive potential, emphasizing that as AI capabilities advance, detecting when models begin mastering manipulation becomes crucial for responsible AI development.