On the Trade-Off Between Transparency and Security in Adversarial Machine Learning

Transparency and security in AI systems can directly conflict: The paper's large-scale evaluation of 181 models found that attackers are significantly more successful when they have information matching the defender's setup, suggesting that complete openness about AI system defenses may actually make those systems more vulnerable to adversarial attacks.

Even limited information about AI defenses can be exploited: The research demonstrates that attackers don't need full transparency to cause damage—simply knowing whether a defender's model uses security protections or not can be enough to substantially reduce the system's security, highlighting how partial information disclosure poses real risks.

"Security through obscurity" may sometimes be justified in adversarial contexts: Contrary to conventional cybersecurity wisdom, the paper's game-theoretic analysis suggests that keeping some details about AI system defenses hidden could be strategically beneficial, at least in adversarial machine learning scenarios where attackers actively try to fool the system.

Game theory reveals hidden trade-offs in responsible AI principles: The paper uses game-theoretic modeling to show that two widely-accepted responsible AI goals—transparency and security—can work against each other, suggesting that organizations may need to make deliberate trade-off decisions rather than pursuing both principles equally.

Responsible AI requires context-specific policy decisions: The findings indicate that a one-size-fits-all approach to transparency won't work; defenders of AI systems need to strategically consider their threat environment and may need to limit transparency in high-risk adversarial settings while maintaining it elsewhere.