Disentangling Algorithmic Bias from Archival Artifacts: A Controlled Audit of Vision-Language Model Valuation in Metropolitan Museum Archives

CLIP models show no statistically significant gender bias in valuing artwork: When researchers tested OpenAI's CLIP and OpenCLIP models on 1,500 artworks from the Metropolitan Museum, they found nearly identical scores for works by male and female artists (p-values of 0.18 and 0.12, respectively), suggesting these particular vision-language models don't systematically undervalue women artists' work.

Archival data quality matters more than algorithm bias in this case: The study found that artwork characteristics like medium, creation era, and aspect ratio explained less than 2% of score variation, indicating that the models' valuations are largely driven by factors in the artwork itself rather than algorithmic gender bias—but this also means the measurement tool is too coarse to detect subtle biases.

"No bias detected" doesn't mean "model is fair": The researchers caution that finding statistical equivalence at a broad level doesn't prove the models are truly fair. The high residual variance suggests these zero-shot evaluation methods operate near their detection limits, meaning smaller or more nuanced biases could still exist but remain hidden.

Distinguishing between algorithm bias and historical bias is crucial: This study demonstrates the importance of separating what a model learned from training data (algorithmic bias) versus what was already present in historical archives (archival bias), a distinction often overlooked in AI audits.