When the Past Misleads: Rethinking Training Data Expansion Under Temporal Distribution Shifts

More historical data doesn't always improve predictions: The study found that while longer training windows initially boost model performance, gains plateau quickly and can actually reverse when the relationship between features and outcomes changes over time—a phenomenon called "concept shift."

Outdated patterns can harm fairness across demographic groups: When concept shifts affect different demographic groups unequally, models trained on expanded historical data produce more biased predictions. For intersectional groups (those with multiple demographic identities), these fairness impacts become even more complex and unpredictable.

Real-world education systems show the problem is practical, not theoretical: An empirical analysis of 12 years of student records from 23 minority-serving community colleges demonstrated that concept shifts are a major cause of performance degradation when training on longer historical periods—suggesting this isn't just a theoretical concern but affects actual high-stakes decisions about student retention.

The type of data change matters significantly: Covariate shifts (changes in feature distributions alone) had minimal impact on model performance, but concept shifts (changes in how features relate to outcomes) substantially degraded predictions, indicating that practitioners need to distinguish between different types of distribution changes when deciding how much historical data to use.