Manifold of Failure: Behavioral Attraction Basins in Language Models

Different AI models have fundamentally different vulnerability patterns: The research found that three major language models—Llama-3-8B, GPT-OSS-20B, and GPT-5-Mini—each have distinct "safety landscapes," meaning they fail in different ways and places. For example, Llama-3-8B showed widespread vulnerabilities across many scenarios, while GPT-5-Mini demonstrated significantly stronger robustness, suggesting that model architecture and training choices have major impacts on safety.

Vulnerabilities cluster into distinct "niches" rather than being randomly distributed: The study discovered up to 370 separate vulnerability categories across models, revealing that unsafe behaviors aren't scattered randomly but instead concentrate in specific regions. This structured understanding could help developers identify and fix entire categories of problems rather than treating each failure as isolated.

Current attack methods only scratch the surface of potential failures: Existing adversarial attack techniques (GCG, PAIR, and TAP) find only discrete examples of failures, but this research's mapping approach revealed the broader topological structure underlying these vulnerabilities—suggesting that current safety testing may be missing systematic patterns of weakness.

Mapping failure regions is as important as preventing them: Rather than only trying to push unsafe outputs back toward safe behavior, the authors argue that comprehensively understanding where and why models fail is essential for responsible AI development, offering a more complete picture of a model's actual safety profile.