Ordinary, Reasonable Chatbots: Do AI Models Track Human Legal Judgments?

AI chatbots can generally replicate human legal reasoning on reasonableness judgments, with the study finding that large language models' responses "generally track those of human participants" across 25 different legally relevant scenarios—suggesting AI could potentially assist in legal contexts, though with important caveats.

AI models produce more uniform responses than humans do, which is concerning because legal reasonableness judgments should reflect the diversity of human perspectives and contexts; the paper notes LLMs generate "more homogeneous responses" compared to the natural variation in how humans evaluate what is "reasonable."

The vagueness of "reasonableness" in law makes it a critical test case for AI reliability, as reasonableness judgments depend heavily on context and implicit understanding—areas where AI consistency might mask important nuances that human judges would naturally capture through lived experience and contextual awareness.

Demographic variation in legal judgments may not be adequately represented in AI systems, as the researchers caution that "reasonableness judgments may vary along demographic lines," raising questions about whether AI trained on broad datasets can fairly represent diverse perspectives that should influence legal decisions.

The rise of "silicon jurors" requires careful scrutiny before deployment, as increasing reliance on AI for legal guidance—whether in research, courtrooms, or policy—demands rigorous testing of whether these systems can truly replicate the contextual judgment and demographic sensitivity that human legal reasoning provides.