Metaethical Perspectives on 'Benchmarking' AI Ethics

There is no universal way to measure AI "ethics": The paper argues, drawing on moral philosophy, that creating a single benchmark for AI ethics is fundamentally impossible—unlike benchmarks for technical tasks like question answering. This challenges the common assumption that we can simply test whether an AI system is "ethical."

"Values" is a more useful framework than "ethics": Rather than debating abstract ethics, the paper recommends focusing on "value alignment"—explicitly identifying what values an AI system should embody and whose values those are. This shift makes the underlying assumptions clearer and more actionable.

Values are relative, and that's actually important: Because values are inherently relative rather than universal, explicitly acknowledging them forces researchers and organizations to be transparent about whose values they're prioritizing in AI development—a crucial step for responsible AI that's often glossed over.

Current approaches to AI ethics may be insufficient for real-world applications: The paper emphasizes that applied, industrial AI research needs better evaluation mechanisms than traditional ethics benchmarks, suggesting that current methods for assessing responsible AI in practice are inadequate.

The field needs new approaches to ensure safe and beneficial AI: Rather than pursuing the impossible goal of universal ethics benchmarks, the paper calls for alternative mechanisms and different research approaches that center on explicit value alignment and stakeholder consideration.