Fooling LLM graders into giving better grades through neural activity guided adversarial prompting

LLM grading systems can be systematically manipulated: Researchers developed a method to fool large language models into assigning significantly higher grades to essays than human evaluators would give, demonstrating that AI evaluation systems contain exploitable vulnerabilities that malicious actors could potentially abuse.

Vulnerabilities transfer across different AI models: The adversarial attacks that worked on one LLM successfully transferred to other models, including commercial closed-source systems like Gemini, suggesting that this bias problem is widespread across the AI industry rather than isolated to specific systems.

Hidden biases stem from AI training design choices: The researchers traced the root cause to "magic words" embedded during the supervised fine-tuning process, showing that vulnerabilities aren't accidental flaws but rather byproducts of how these systems are currently built and trained.

Simple fixes can substantially reduce bias: The paper demonstrates that minor modifications to the chat templates used during AI training can "drastically reduce" these biases, suggesting that responsible AI deployment requires attention to training methodology details that are often overlooked.

Systematic bias detection is possible and necessary: The research provides a method to proactively identify hidden biases in AI systems before deployment, offering a practical pathway for organizations to audit their AI evaluation tools for vulnerabilities in critical applications like grading, hiring, or lending decisions.