From Model Performance to Claim: How a Change of Focus in Machine Learning Replicability Can Help Bridge the Responsibility Gap

The "Responsibility Gap" is a real problem in ML research: Machine learning scientists are often far removed from where their work is actually applied, making it difficult to hold them accountable when their research causes harm through misuse or misinterpretation.

Two types of replicability need to be distinguished: The paper identifies "model performance replicability" (reproducing the same technical results) versus "claim replicability" (ensuring the broader claims made about what a model does are accurate and reproducible). These are currently conflated in discussions about ML research quality.

Focusing on claim replicability can bridge the accountability gap: By shifting attention to whether the claims scientists make about their models are actually replicable and defensible, researchers can be held responsible for non-replicable claims that are "prone to eliciting harm due to misuse and misinterpretation."

Claim replicability is fundamentally a social responsibility issue, not just a technical one: The paper argues that implementing claim replicability is "more of a social project than a technical one," suggesting that fixing this problem requires changes to how the ML community communicates and validates research findings, not just better code and data sharing.

This approach makes accountability actionable for ML scientists: By reconceptualizing replicability around claims rather than just model performance, the paper argues researchers can take concrete responsibility for the accuracy and clarity of what they communicate about their work's capabilities and limitations.