Supporting Human-AI Collaboration in Auditing LLMs with LLMs

Human-AI collaboration can uncover diverse AI failures: In user studies with AdaTest++, participants identified failure modes across 26 different topics when auditing GPT-3 and Azure's sentiment analysis model, including both previously documented issues and under-reported problems, demonstrating that combining human and AI strengths finds more comprehensive issues than either approach alone.

Sensemaking and communication are critical for effective auditing: The research emphasizes that how humans and AI systems communicate and make sense of findings together is essential to auditing—the tool was specifically designed to support human reasoning processes like hypothesis formation and testing, not just automate failure detection.

Current LLM auditing tools have significant gaps: The paper identifies that existing auditing approaches relying solely on humans or AI are insufficient for catching the range of biases and irresponsible behaviors in large language models, suggesting that responsible AI deployment requires more sophisticated collaborative auditing methods.

Generative LLMs can assist in auditing other LLMs: By leveraging a generative language model to power the auditing tool itself, the research shows that LLMs can be repurposed to help identify failures and biases in other AI systems—though human oversight remains essential for interpreting and validating findings.

Responsible AI requires rigorous, ongoing testing: The paper underscores that as language models become more widespread in society, systematic auditing processes are not optional but crucial to preventing harm at scale, particularly for high-stakes applications.