Mirage: Unveiling Hidden Artifacts in Synthetic Images with Large Vision-Language Models

AI detection systems are falling behind image generation technology: Current state-of-the-art detection methods fail to identify many AI-generated images in the Mirage dataset, even though humans can still spot visible artifacts in these images, revealing a growing gap between synthetic image quality and detection capabilities.

Large Vision-Language Models show promise but have clear limitations: While LVLMs (like GPT-4V) can effectively detect AI-generated images when visible artifacts are present, their performance significantly declines when analyzing images without obvious flaws, suggesting they cannot be fully relied upon as replacements for human judgment.

"Visible artifacts" are not a reliable detection standard: The research highlights that relying on human-visible imperfections as a detection method is problematic, since advanced generators are increasingly producing images without obvious artifacts—meaning future synthetic images may be undetectable by both current AI systems and human observers.

Explainability matters for AI detection tools: The paper emphasizes the value of using LVLMs specifically because they can provide explanations for their detection decisions, which is important for responsible AI deployment where stakeholders need to understand why an image is flagged as synthetic, not just that it is.

We need better benchmarks for real-world scenarios: The introduction of the Mirage dataset addresses a gap in existing benchmarks by focusing on harder detection cases, suggesting that current evaluation methods may be giving a false sense of security about our ability to identify synthetic media in practice.