Cannot or Should Not? Automatic Analysis of Refusal Composition in IFT/RLHF Datasets and Refusal Behavior of Black-Box LLMs

Refusals serve dual purposes in AI safety: The paper distinguishes between "cannot" refusals (where models lack capability, like inability to perform complex math) and "should not" refusals (where models decline for safety reasons, like refusing to help with illegal activities). Current evaluation methods focus too heavily on the latter, missing important capability limitations.

Training data shapes refusal behavior: Refusals are primarily learned during post-training phases like instruction fine-tuning (IFT) and reinforcement learning from human feedback (RLHF), meaning the quality and composition of training data directly influences how and when AI systems refuse requests—a critical lever for responsible AI development.

Existing tools for auditing AI refusals are inadequate: The paper identifies that current taxonomies and datasets for understanding refusals are incomplete and lack practical tools for analyzing black-box LLM outputs, making it difficult for organizations to systematically audit whether their AI systems are refusing appropriately.

A comprehensive classification framework is now available: The researchers created a 16-category taxonomy with over 8,600 human-annotated real examples and 8,000 synthetic examples per category, providing the first systematic tool for precisely categorizing and understanding different types of LLM refusals.

Better refusal auditing enables safer AI systems: By enabling precise measurement and analysis of refusal patterns, this framework allows developers to strategically adjust how their models refuse requests, supporting the creation of more reliable and trustworthy AI systems.