Can LLMs Rank? A Tale of Triads and Triage

LLMs show inconsistent ranking abilities across different tasks: When tested on high-stakes decisions like allocating housing for homeless individuals and emergency department triage, three leading LLMs demonstrated "considerably different performance profiles," suggesting that an LLM's reliability cannot be assumed to transfer across different prioritization contexts.

Two independent consistency measures are both necessary: The paper identifies intra-run consistency (whether an LLM makes circular or contradictory judgments within a single ranking attempt, measured by the coefficient of consistency ζ) and inter-run variability (whether the LLM produces different rankings across multiple attempts, measured by Kendall's τ) as separate reliability indicators that should both be checked before deploying an LLM for consequential decisions.

Practitioners need a pre-deployment reliability check: Before using an LLM to rank people for scarce resources, practitioners can use the coefficient of consistency ζ as a "cheap, model-free measure" to identify problematic circular reasoning patterns—providing an accessible way to flag when an LLM's judgments may be too unreliable to trust without requiring deep technical expertise.

Pairwise comparison is a theoretically sound but practically risky approach: While social choice theory supports breaking down large ranking tasks into pairwise comparisons to reduce cognitive burden, this paper reveals that LLMs' performance on these pairwise judgments varies significantly, meaning this approach doesn't automatically solve the reliability problem for high-stakes decisions.