Adversarial Pragmatics for AI Safety Evaluation: A Diagnostic Framework and Seed Benchmark for Language-Mediated Control

Current safety benchmarks hide the real reasons for failures: Existing pass/fail evaluations obscure whether language model failures stem from capability limits, unclear policies, conflicting instructions, or inconsistent evaluator judgments—making it impossible to diagnose and fix underlying problems.

Language models struggle with pragmatic ambiguity in safety-critical situations: The framework identifies specific linguistic challenges (embedded commands, indirect requests, scope ambiguity, and quoted text) where models may misinterpret instructions in ways that bypass safety policies, representing a distinct safety concern beyond simple refusal failures.

AI judges can miss safety-relevant edge cases: When an LLM judge evaluated its own outputs on this benchmark, it "missed the safety-relevant minority classes"—suggesting that automated evaluation systems may systematically overlook the unusual but important failure modes that matter most for safety.

Safety evaluation requires separating multiple distinct factors: The framework analytically separates task success, policy compliance, risk level, refusal behavior, and evaluator confidence rather than collapsing them into single labels, enabling more precise diagnosis of where safety systems actually break down.

This tool is designed for diagnosis, not deployment decisions: The authors explicitly state the benchmark is intended for diagnostic understanding rather than vendor ranking or certification, cautioning against using it to make high-stakes deployment or comparison decisions without additional evidence.