Quantifying Risk Propensities of Large Language Models: Ethical Focus and Bias Detection through Role-Play

LLMs exhibit measurable "risk personalities" that vary by domain: The researchers adapted cognitive science tools (the DOSPERT scale) to systematically evaluate how different large language models approach risky decisions, revealing that these AI systems have consistent patterns in their decision-making tendencies that differ across ethical, financial, social, and other domains.

Ethical decision-making in AI can be quantitatively assessed: The study introduces a new measurement tool called the Ethical Decision-Making Risk Attitude Scale (EDRAS) specifically designed to evaluate how LLMs handle ethical dilemmas, enabling more rigorous and comparable testing of AI systems' moral reasoning compared to previous informal approaches.

Role-playing scenarios effectively expose hidden biases: By combining risk assessment scales with role-playing exercises, the researchers developed a method that reveals systematic biases in LLMs toward different groups—biases that might not be apparent through standard testing and that could perpetuate unfair treatment if deployed without detection.

Mainstream LLMs show quantifiable systematic biases: The evaluation of multiple popular LLMs uncovered and measured specific biases in how these systems treat different groups, providing concrete evidence that current widely-used AI systems have fairness problems that need addressing before broader deployment.

Standardized evaluation tools are essential for AI safety: This work demonstrates that applying rigorous, systematic measurement approaches from cognitive science to AI can help identify risks and biases early, supporting the development of more trustworthy and reliable AI systems before they're deployed at scale.