AI2030 Responsible AI Tools Marketplace

Key Takeaways

Explanations alone don't guarantee better decisions: Despite the widespread belief that algorithmic explanations improve fairness and responsible decision-making, this study found that counterfactual explanations (which show how a risk assessment score would change under different circumstances) did not alter judges' actual decisions in pretrial cases.
Judges initially misunderstood the explanations as real rather than hypothetical: The eight sitting judges in the study first interpreted the counterfactual scenarios as actual changes to defendants' circumstances, rather than understanding them as "what-if" hypothetical scenarios, suggesting that the format of these explanations may be fundamentally confusing to users.
Judges rejected the explanations based on their professional role: Once judges understood that counterfactuals were hypothetical, they deliberately ignored them, reasoning that their job is to make decisions about the actual defendant in front of them, not imaginary versions of that person—raising questions about whether this type of explanation is even appropriate for legal decision-making contexts.
The path to better human-algorithm collaboration is more complex than expected: The study reveals "unexpected ways in which people respond to algorithms and explanations," indicating that simply adding more information or transparency to algorithmic tools may not solve fairness problems and could create new challenges in how humans and algorithms work together.
One-size-fits-all explanations may not work across different professional contexts: The findings suggest that explanation methods designed in research settings may not translate effectively to real-world professional decision-making, where users have established practices and role-specific reasoning that can override algorithmic guidance.

Abstract

Many researchers and policymakers have expressed excitement about algorithmic explanations enabling more fair and responsible decision-making. However, recent experimental studies have found that explanations do not always improve human use of algorithmic advice. In this study, we shed light on how people interpret and respond to counterfactual explanations (CFEs) -- explanations that show how a model's output would change with marginal changes to its input(s) -- in the context of pretrial risk assessment instruments (PRAIs). We ran think-aloud trials with eight sitting U.S. state court judges, providing them with recommendations from a PRAI that includes CFEs. We found that the CFEs did not alter the judges' decisions. At first, judges misinterpreted the counterfactuals as real -- rather than hypothetical -- changes to defendants. Once judges understood what the counterfactuals meant, they ignored them, stating their role is only to make decisions regarding the actual defendant in question. The judges also expressed a mix of reasons for ignoring or following the advice of the PRAI without CFEs. These results add to the literature detailing the unexpected ways in which people respond to algorithms and explanations. They also highlight new challenges associated with improving human-algorithm collaborations through explanations.

Accepted at HCOMP '22 and at CHI'22 Workshop on Human-Centered Perspectives in Explainable AI (HCXAI)

Key Takeaways

Explanations alone don't guarantee better decisions: Despite the widespread belief that algorithmic explanations improve fairness and responsible decision-making, this study found that counterfactual explanations (which show how a risk assessment score would change under different circumstances) did not alter judges' actual decisions in pretrial cases.
Judges initially misunderstood the explanations as real rather than hypothetical: The eight sitting judges in the study first interpreted the counterfactual scenarios as actual changes to defendants' circumstances, rather than understanding them as "what-if" hypothetical scenarios, suggesting that the format of these explanations may be fundamentally confusing to users.
Judges rejected the explanations based on their professional role: Once judges understood that counterfactuals were hypothetical, they deliberately ignored them, reasoning that their job is to make decisions about the actual defendant in front of them, not imaginary versions of that person—raising questions about whether this type of explanation is even appropriate for legal decision-making contexts.
The path to better human-algorithm collaboration is more complex than expected: The study reveals "unexpected ways in which people respond to algorithms and explanations," indicating that simply adding more information or transparency to algorithmic tools may not solve fairness problems and could create new challenges in how humans and algorithms work together.
One-size-fits-all explanations may not work across different professional contexts: The findings suggest that explanation methods designed in research settings may not translate effectively to real-world professional decision-making, where users have established practices and role-specific reasoning that can override algorithmic guidance.

Abstract

Accepted at HCOMP '22 and at CHI'22 Workshop on Human-Centered Perspectives in Explainable AI (HCXAI)

"If it didn't happen, why would I change my decision?": How Judges Respond to Counterfactual Explanations for the Public Safety Assessment

Key Takeaways

Abstract

Paper Info

Responsible AI Topics

Links

"If it didn't happen, why would I change my decision?": How Judges Respond to Counterfactual Explanations for the Public Safety Assessment

Key Takeaways

Abstract

Paper Info

Responsible AI Topics

Links