Inserting Faces inside Captions: Image Captioning with Attention Guided Merging

Current image captioning systems struggle with identifying and naming people: Traditional image captioning models are inefficient and biased when it comes to retrieving people's names, which limits their usefulness for accessibility and photo organization purposes.

A new dataset addresses the gap for public figures: The AstroCaptions dataset was created specifically to help train models on images of thousands of public figures that are complex for standard models to identify, providing a resource for improving this capability.

A novel post-processing method successfully inserts names into captions: The researchers developed a technique using explainable AI tools that can insert identified people's names into image captions after the fact, with up to 93.2% accuracy in detecting and naming persons in images.

This approach reduces AI hallucinations and improves caption quality: By grounding captions in actual identified people rather than relying solely on the model's generation, the method reduces false information and improves multiple quality metrics (BLEU, ROUGE, CIDEr, and METEOR scores) across different captioning models.

Explainability enables more reliable AI systems: The use of explainable AI tools to guide the name-insertion process demonstrates how transparency in AI decision-making can lead to more trustworthy and accurate outputs for real-world applications like photo accessibility.