Understanding Endogenous Data Drift in Adaptive Models with Recourse-Seeking Users

Feedback loops between users and AI models create unintended consequences: When people adapt their behavior to meet AI system requirements (seeking "recourse"), this changes the data the model learns from. The model then retrains on this shifted data, which further changes what users need to do—creating a self-reinforcing cycle that wasn't anticipated during initial model development.

AI systems can inadvertently raise standards over time, making it harder for people to succeed: The research found that as users adapt their features to pass AI decision-making systems, models tend to shift toward increasingly higher decision thresholds. This means people must work harder and harder to achieve positive outcomes, even though the underlying system wasn't explicitly designed to do this.

Current AI practices assume static conditions that don't reflect reality: Most deep learning models are built assuming the world stays the same between training and deployment. However, real-world systems interact with people who actively respond to decisions, violating this core assumption and potentially leading to unreliable or unfair outcomes over time.

Proposed solutions can reduce costs and improve reliability: Two methods—Fair-top-k and Dynamic Continual Learning—showed promise in reducing the burden on users ("recourse cost") and making models more robust when facing these adaptive behaviors, suggesting practical ways to address the problem.

Algorithmic decision-making can unintentionally create economic barriers: The findings connect to economic theory, showing how AI systems can inadvertently reinforce higher standards and generate barriers that weren't explicitly programmed but emerge from the interaction between users and algorithms.