ArchiveGPT: A human-centered evaluation of using a vision language model for image cataloguing

Humans cannot reliably detect AI-generated catalogue descriptions: Both expert and non-expert participants performed above chance level at distinguishing AI-written from human-written descriptions, but significantly underestimated their own ability to do so, suggesting people may have false confidence in their capacity to spot AI errors in specialized domains.

AI hallucinations and OCR errors undermine quality: The vision language model (InternVL2) produced descriptions with factual errors and misreadings of text on archival materials, limiting perceived quality—though descriptions rated as more accurate were paradoxically harder for humans to identify as AI-generated, indicating that human review remains essential.

Expert archivists resist AI adoption despite technical capability: Experts showed notably lower willingness to adopt AI tools compared to non-experts, driven not by performance concerns but by concerns about preservation responsibility and their professional duty to ensure accuracy, highlighting that technical feasibility alone doesn't determine real-world adoption.

AI works best as a draft assistant, not autonomous cataloguer: The research supports a collaborative workflow where AI generates initial descriptions for human review and refinement, rather than replacing human expertise—particularly important in specialized fields like archaeology where domain knowledge is critical for accuracy.

Specialized domains require human oversight: The study demonstrates that out-of-the-box AI models need human verification to ensure quality in specialized cataloguing contexts, suggesting that general-purpose AI tools cannot be deployed without expert validation in archival and museum settings.