Measuring What Matters: Connecting AI Ethics Evaluations to System Attributes, Hazards, and Harms

Most AI ethics measures focus on only four principles (fairness, transparency, privacy, and trust) out of 11 identified principles, leaving other important ethical considerations like accountability and human autonomy largely unmeasured in the computing literature.

Current evaluation practices are fragmented and incomplete: The analysis of nearly 800 measures found that most assess only individual model or output components rather than examining how different parts of an AI system interact, meaning evaluations may miss harms that emerge from the system working as a whole.

Measures often fail to connect to real-world harms: Few measures are grounded in where actual harm occurs or provide clear guidance on what acceptable performance thresholds should be, making it difficult for organizations to translate ethical evaluations into meaningful protections for affected people.

A narrow range of hazards is considered for each type of harm: The research reveals that evaluation practices typically only examine a limited set of potential risks for each harm category, potentially overlooking important failure modes and unintended consequences.

Better measurement frameworks could improve oversight and industry practice: By connecting ethical measures to specific system components, hazards, and harms, the authors argue that organizations can develop more actionable evaluation practices and regulators can establish more effective oversight of AI systems.