A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applications

Automated measurement of AI harms is now feasible: The paper demonstrates that responsible AI metrics for large language models can be systematically measured using automated frameworks, moving beyond ad-hoc testing toward more consistent and scalable evaluation methods.

Advanced AI systems can audit themselves: The framework leverages state-of-the-art LLMs like GPT-4 to measure harms in other LLMs, suggesting that AI systems themselves can be tools for identifying problematic outputs—though this approach requires careful validation and human oversight.

Multiple responsible AI principles can be violated in different ways: The case studies reveal that different LLMs violate various responsible AI principles in distinct patterns, indicating that no single model or product uniformly fails or succeeds across all harm categories, and that tailored evaluation is necessary.

Human expertise remains essential: The framework is explicitly designed to work "alongside domain-specific sociotechnical expertise," emphasizing that automated measurement alone is insufficient—subject matter experts and affected communities must remain involved in defining and interpreting what constitutes harm.

This framework enables scalable accountability: By providing a replicable method for measuring responsible AI harms, the framework could help organizations systematically track whether their LLM products are becoming safer over time and support more transparent reporting to users and regulators.