SAGE: A Generic Framework for LLM Safety Evaluation

Single-turn safety tests miss real-world risks: Current AI safety benchmarks evaluate models in isolated interactions, but SAGE's multi-turn conversations revealed that "harm increases with conversation length," meaning risks that go undetected in standard tests can emerge through extended dialogue—a critical gap as LLMs are deployed in healthcare, financial advice, and other high-stakes applications.

User personality and context dramatically change model behavior: The framework tested models against adversarial agents with different personality types (based on the Big Five personality model) and found that "model behavior varies significantly when exposed to different user personalities and scenarios," demonstrating that the same model can behave very differently depending on conversational context—something generic safety policies fail to capture.

There's a tradeoff between safety and usefulness: The research found that "some models minimize harm via high refusal rates that reduce usefulness," meaning overly cautious safety measures can make AI systems less helpful. This highlights the need to balance genuine safety with maintaining practical functionality.

Safety policies are sensitive and context-dependent: When researchers tightened a child-focused sexual content policy, they found it "substantially increases measured defects across applications," showing that safety thresholds aren't universal—what counts as "safe" depends heavily on specific application contexts and policy choices.

One-size-fits-all safety evaluation is insufficient: The paper demonstrates that effective AI safety requires "adaptive, policy-aware, and context-specific testing" rather than generic benchmarks, suggesting organizations deploying AI need customized evaluation frameworks tailored to their specific use cases and potential harms.