Confronting Label Indeterminacy in Automated Bail Decisions

The "Missing Data" Problem in Criminal Justice: When bail is denied, we never know if the defendant would have actually appeared in court, creating what researchers call "label indeterminacy." This means historical data used to train bail prediction systems is fundamentally incomplete, yet these systems are still being deployed to make real decisions affecting people's freedom.

How You Handle Missing Information Matters More Than You'd Think: The paper found that the choice of how to deal with this missing data "sometimes even more so than the choice of model itself" influences what the AI system predicts. This means that subjective decisions made during system development—not just the algorithm itself—can dramatically change outcomes for defendants.

All Approaches Rely on Unverifiable Assumptions: The researchers tested five different methods for handling the missing data, and found that "each method relies on unverifiable assumptions." There is no objectively "correct" way to solve this problem, which raises serious questions about whether these systems can be legitimately used in legal contexts where decisions must be defensible and fair.

Risk of Reinforcing Existing Bias: Because past bail denials shape the training data in ways we can't fully verify, automated systems risk creating feedback loops that perpetuate historical biases in bail decisions rather than improving fairness.

Explainability Doesn't Solve the Core Problem: Even when researchers used tools to explain how the AI makes decisions, the fundamental uncertainty about what the data actually represents remained—suggesting that transparency alone cannot address the legitimacy concerns with using such systems in bail decisions.