ViT-EnsembleAttack: Augmenting Ensemble Models for Stronger Adversarial Transferability in Vision Transformers

Vision Transformers require specialized attack strategies: This is the first ensemble-based adversarial attack method specifically designed for Vision Transformers (ViTs), addressing a gap in security research since previous work focused primarily on traditional convolutional neural networks.

Adversarial augmentation of models improves attack transferability: By artificially modifying surrogate models through techniques like attention head dropping and feature mixing before generating attacks, the method achieves significantly stronger transferability—meaning attacks crafted on one model are more likely to fool other models in real-world deployment scenarios.

Ensemble approaches amplify adversarial vulnerabilities: Combining multiple augmented models to generate attacks proves substantially more effective than single-model approaches, suggesting that the diversity created through model augmentation helps attacks generalize better across different AI systems.

Automated optimization enhances attack effectiveness: Using Bayesian optimization to fine-tune parameters for model augmentation strategies, combined with automatic reweighting and step size adjustment modules, demonstrates that systematic tuning of attack methods can significantly increase their success rates.

Implications for AI security testing: These findings highlight the importance of testing AI systems—particularly Vision Transformers—against sophisticated ensemble-based attacks during development, as such attacks may be more effective at finding vulnerabilities than simpler, single-model attack methods.