A moving target in AI-assisted decision-making: Dataset shift, model updating, and the problem of update opacity

ML systems degrade over time due to "dataset shift" — when the real-world data an AI system encounters changes from what it was trained on — requiring regular updates to maintain performance, but these updates create a previously overlooked problem for decision-making transparency.

"Update opacity" is a new form of AI opacity: when organizations update ML models, users (including decision-makers and those affected by decisions) often cannot understand how or why the system's reasoning or behavior has changed, making it difficult to trust or audit the system's decisions.

Existing solutions to AI transparency are insufficient: Current approaches to addressing the "black box problem" in AI — like explainability tools — are largely not equipped to handle the specific challenges posed by model updates, leaving a significant gap in responsible AI practices.

Update opacity creates both epistemic and safety risks: The paper identifies distinctive concerns including the inability to verify whether updates improved fairness or introduced new biases, and uncertainty about whether updated systems will behave as expected in critical decision-making contexts (healthcare, criminal justice, lending, etc.).

Proposed solutions like "dynamic model reporting" and "update compatibility" exist but have limitations: While strategies to address update opacity are being developed, each carries its own risks or practical constraints, indicating that organizations need tailored approaches and that further research is needed to protect decision-making integrity.