Street-Level AI: Are Large Language Models Ready for Real-World Judgments?

LLMs show dangerous inconsistency in high-stakes decisions: The same LLM produces different prioritization results on different runs, and different LLMs disagree with each other when asked to allocate homeless services—a critical problem when AI systems are used to decide who receives scarce social resources.

Current AI systems don't reliably match established fairness standards: LLM judgments are inconsistent with existing vulnerability scoring systems that have been socially and politically determined for resource allocation in homelessness services, suggesting AI could undermine established principles of local justice.

AI may seem fair in simple tests but fail in real-world complexity: While LLMs showed qualitative consistency with lay human judgments in pairwise comparisons, this apparent alignment masked deeper problems when tested on actual cases requiring real resource allocation decisions.

Real-world deployment of LLMs for "street-level" decisions is premature: The paper directly challenges the readiness of current AI systems to replace or assist human bureaucrats who make consequential decisions about benefit allocation, calling into question recent enthusiasm for AI integration in these domains.

Using real data reveals problems that thought experiments miss: By testing LLMs on actual confidential cases from people experiencing homelessness (rather than hypothetical scenarios), the researchers uncovered inconsistencies that ethical analyses based on abstract moral dilemmas had not identified.