Lessons From Red Teaming 100 Generative AI Products

Red teaming requires understanding real-world deployment contexts. The first lesson emphasizes that testing AI systems in isolation isn't enough—researchers must understand how and where these systems will actually be used, since the same AI product poses different risks in different applications.

You don't need technical expertise to find AI vulnerabilities. Contrary to assumptions that breaking AI systems requires deep machine learning knowledge, the paper finds that non-technical attacks (like carefully crafted prompts) can be just as effective as sophisticated technical exploits, making AI security a broader responsibility than just engineers.

Red teaming and safety benchmarking serve different purposes. The paper clarifies that red teaming (actively searching for failures) is distinct from benchmarking (measuring performance on standardized tests), and organizations shouldn't confuse one for the other when evaluating AI safety.

Human judgment remains irreplaceable in AI safety testing. While automation can efficiently test many scenarios, human red teamers are essential for identifying novel risks and understanding context—suggesting that responsible AI requires sustained human oversight rather than fully automated solutions.

AI harms are widespread but hard to quantify. The paper notes that responsible AI concerns like bias and misuse are pervasive across products, yet measuring and comparing these harms remains difficult, highlighting a key challenge in making AI safety practices consistent and verifiable across the industry.