Explainable Black-Box Attacks Against Model-based Authentication

ML-based authentication systems are vulnerable to attack: Researchers successfully fooled state-of-the-art face authentication systems in under 130 queries and host authentication systems in under 100 queries, demonstrating that machine learning models used for identity verification can be systematically compromised.

Explainable AI tools can be weaponized: The paper shows that Explainable-AI (XAI) techniques—tools designed to make AI systems more transparent and trustworthy—can actually be exploited by attackers to understand how authentication systems make decisions and find ways around them.

Current ML authentication lacks sufficient security vetting: While machine learning-based authentication offers convenience through automated identity verification, the paper reveals these systems have not been adequately tested against sophisticated attacks before being deployed, creating a significant security gap.

Attackers don't need full system access: The "blind attack" approach used in this research required only the ability to query the system repeatedly, not access to internal model details, suggesting real-world authentication systems could be vulnerable to attacks from outside actors.

Security and explainability may create a tradeoff: The findings suggest that making AI systems more interpretable and explainable—a key goal in responsible AI—could inadvertently make them easier to attack, highlighting a potential tension between transparency and security in AI system design.