Chain-of-Jailbreak Attack for Image Generation Models via Editing Step by Step

Current safety filters are vulnerable to step-by-step manipulation: The Chain-of-Jailbreak attack successfully bypassed safety guardrails in major image generation models (GPT-4V, GPT-4o, Gemini 1.5, and Gemini 1.5 Pro) over 60% of the time by breaking harmful requests into multiple sub-queries, compared to only 14% success with other jailbreaking methods.

Iterative editing poses a unique safety challenge: Unlike single-prompt attacks, the CoJ method exploits the image editing functionality by having models progressively modify images based on sequential instructions, revealing that current safety measures may not adequately monitor multi-step generation processes.

Comprehensive testing reveals systemic vulnerabilities: The researchers tested their attack across nine different safety scenarios using a dataset called CoJ-Bench, suggesting that safety failures are not isolated incidents but represent broader weaknesses across multiple categories of harmful content (abusive, violent, pornographic material).

Defensive measures are being developed: The paper proposes a prompting-based defense method to enhance model safety against these attacks, indicating that researchers are actively working on solutions, though the effectiveness of these defenses requires further evaluation.

Popular commercial models need urgent safety improvements: The vulnerability affects widely-used, commercially deployed systems that many people rely on for content creation, highlighting the need for developers to strengthen safeguards before these models are used at scale.