Searching for "Harmful Refusal": A Psychometric Audit of an AI Safety Benchmark

Safety benchmarks may mask important differences between AI models: The paper finds that models with similar overall safety scores can have "very different attribute profiles," meaning two models that appear equally safe by standard metrics may actually behave quite differently when refusing harmful requests.

Current benchmarks don't reliably measure what they claim to measure: Using psychometric testing methods borrowed from psychology, researchers found that HarmBench—a major AI safety dataset—does not actually measure a single, consistent attribute like "harmful refusal" as intended, suggesting the benchmark may not be capturing what developers think it captures.

Developer differences may be confounding safety evaluations: The analysis found "differential item functioning" where AI models from different companies with supposedly equal refusal abilities actually scored differently on the same test items, raising concerns that benchmark results may reflect developer-specific factors rather than true safety capabilities.

Aggregating diverse harms into one score obscures critical details: The paper demonstrates that HarmBench "collapses distinct harm behaviors" into a single overall score, which could hide important safety gaps—for example, a model might refuse certain types of harmful requests while failing on others, but this distinction gets lost in the final number.

Psychometric rigor is needed in AI safety evaluation: By applying construct validity frameworks from psychology research, this work highlights that safety benchmarks need more rigorous validation to ensure they're actually measuring meaningful, single attributes rather than conflating multiple different capabilities.