Can MLLMs Read the Room? A Multimodal Benchmark for Verifying Truthfulness in Multi-Party Social Interactions

Current AI systems struggle significantly with detecting deception in social settings: Even state-of-the-art models like GPT-4o show substantial performance gaps when asked to distinguish truth from falsehood in multi-party conversations, suggesting that advanced AI cannot yet reliably assess truthfulness in realistic human interactions.

AI models fail to integrate visual and verbal information effectively: The research reveals that multimodal AI systems don't adequately ground language understanding in visual social cues (like facial expressions, body language, and gestures), which are critical for humans to detect deception—a key limitation for trustworthy AI deployment in social contexts.

Alignment training may be making AI systems overly cautious: The analysis indicates that safety measures designed to make AI systems more trustworthy may be causing them to be "overly conservative," potentially limiting their ability to make nuanced judgments about complex social situations.

Deception detection requires understanding dynamic, multi-party interactions: The benchmark uses the social deduction game Werewolf to create realistic scenarios with synchronized video and text, demonstrating that truthfulness assessment isn't just about analyzing isolated statements but requires understanding complex group dynamics and evolving conversations.

We need new approaches to build more socially intelligent AI: The findings highlight an "urgent need for novel approaches" to developing AI systems that can perceive and respond appropriately to social cues, suggesting current methods are insufficient for AI systems that will interact meaningfully with humans in social settings.