Stranded Credentials: Keeping Online Reputation Systems Informative in the AI Era

Credentials remain surprisingly informative during AI transitions: Despite concerns that generative AI would render quality credentials worthless, medals on Kaggle—a data science competition platform—retained most of their predictive power through the AI era, suggesting that reputation systems can survive technological disruption better than feared.

Recency matters far more than lifetime achievement: Across nearly 450,000 competition participations, a medal's ability to predict future performance "sits almost entirely in its first year," meaning recent accomplishments are vastly more informative than older credentials for assessing current capability—a finding that applies both before and after the AI transition.

Platforms may inadvertently hide valuable information: Kaggle's official display of lifetime achievement tiers discards up to one-sixth of the predictive information contained in medals. A simple recency-weighted index would explain AI-era performance about 13% better than the current system while identifying stronger average performers.

Some credential devaluation predates AI: Approximately half of the decline in one medal format's informativeness occurred because the platform was phasing out that format before AI arrived, suggesting that institutional decisions—not just technological change—can erode credential value.

Simple transparency improvements could restore trust: Adding a recent-performance summary alongside lifetime credentials would help buyers make better-informed decisions without requiring complex algorithmic changes, offering a practical path to maintaining informative reputation systems.