Assessing Human Rights Risks in AI: A Framework for Model Evaluation

Human rights frameworks can be systematically applied to AI evaluation: The paper demonstrates that principles from the Universal Declaration of Human Rights and UN Guiding Principles on Business and Human Rights can be translated into concrete computational methods for assessing AI risks, moving beyond abstract concerns to measurable evaluation criteria.

Context matters critically for AI harm assessment: The authors emphasize that human rights risks cannot be evaluated in a vacuum—the same AI model poses different risks depending on where and how it's deployed. Their case study of large language models in political journalism shows that real-world context is essential for grounded claims about potential harms.

A three-part evaluation approach can identify and measure rights risks: The framework consists of (1) identifying tasks likely to pose human rights risks, (2) designing metrics to measure scope, scale, and likelihood of those risks, and (3) analyzing specific rights implications—providing a structured methodology that organizations can adapt to their own AI systems.

AI systems can threaten fundamental rights like access to information and freedom of thought: The political journalism case study reveals concrete risks to rights beyond typical AI concerns like discrimination, including how language models might affect citizens' ability to access reliable information and form independent thoughts—highlighting that human rights impacts extend beyond individual fairness to societal-level consequences.

Systematic auditing of AI requires moving beyond one-size-fits-all metrics: Rather than applying generic fairness benchmarks, the paper argues that responsible AI evaluation must be tailored to specific deployment contexts and the particular human rights at stake in each situation.