Under the Influence: Quantifying Persuasion and Vigilance in Large Language Models

Persuasion and vigilance are separate skills in AI systems: The research found that LLMs' ability to solve problems well does not automatically translate to detecting when they're being misled by other AI systems—even when explicitly warned that deception is possible. This suggests that being good at a task doesn't make an AI system better at protecting itself from bad advice.

AI systems show signs of effort when suspicious but still fail: While LLMs used more computational resources (tokens) to reason through malicious advice compared to benevolent advice, this increased caution didn't actually prevent them from being persuaded into making wrong decisions. This indicates AI systems may appear to be thinking critically without actually being resistant to manipulation.

Current LLMs lack robust defenses against adversarial influence: The study demonstrates that LLMs can be persuaded to take actions that lead to failure, raising concerns about their reliability as advisors in high-stakes decisions where human safety or important outcomes are at risk. This is particularly concerning as these systems are increasingly deployed in areas requiring trustworthy guidance.

The relationship between reasoning ability and trustworthiness is not straightforward: The dissociation between puzzle-solving performance and the ability to maintain vigilance suggests that simply making AI systems "smarter" at their primary task won't automatically make them more resistant to manipulation or better judges of information quality.