LLM Voting: Human Choices and AI Collective Decision Making

LLMs exhibit systematic biases in voting that differ from human preferences: GPT-4 and LLaMA-2 demonstrated voting patterns that don't naturally align with human choices, suggesting these models cannot simply replace or directly represent human decision-making in democratic contexts.

Presentation matters significantly: The order in which voting options are presented influences how LLMs vote, indicating they are susceptible to positional bias—a finding with important implications for any system that might use LLMs to aggregate preferences or make collective decisions.

Persona adjustments can help but aren't a complete solution: While varying the persona assigned to an LLM reduced some biases and improved alignment with human voting patterns, this approach only partially addresses the underlying issues, suggesting no simple technical fix exists.

There's a fundamental trade-off between diversity and accuracy: The paper identifies a tension where LLMs configured to match human voting accuracy tend to produce less diverse collective outcomes, raising concerns about whether AI-assisted voting systems could inadvertently suppress minority viewpoints or reduce the range of perspectives considered.

Explainability techniques like Chain-of-Thought don't guarantee better decisions: Despite their potential value for transparency, asking LLMs to explain their reasoning (Chain-of-Thought prompting) did not improve voting prediction accuracy, cautioning against assuming that more "transparent" AI systems will automatically make better choices.