Forbidden Science: Dual-Use AI Challenge Benchmark and Scientific Refusal Tests

AI models show dramatically different safety approaches: Claude-3.5-sonnet refused 73% of controlled substance queries while Mistral answered all of them, demonstrating that there is no industry standard for how AI systems should handle potentially dangerous information—a critical gap as these tools become more widely used.

Safety mechanisms are vulnerable to prompt manipulation: When researchers tested the same queries with slight variations, response consistency dropped from 85% with a single prompt to 65% with five variations, suggesting that determined users may be able to bypass safety restrictions through rewording, which has real implications for how well these safeguards actually work in practice.

The "over-censorship" problem is real and measurable: The paper identifies a genuine tension between blocking harmful content and restricting legitimate scientific research, highlighting that overly restrictive AI systems could impede important work in fields like chemistry, medicine, and public health—suggesting that responsible AI safety requires nuance, not just blanket refusals.

Open benchmarks are essential for accountability: By releasing a public dataset and testing framework, the researchers enable independent verification of AI safety claims and allow the research community to track whether companies are actually improving their safety mechanisms over time, rather than relying on companies' own assessments.

Current safety implementations lack consistency and transparency: The wide variation in refusal rates across models suggests companies are using different (and often undisclosed) approaches to safety, making it difficult for users to understand what they can rely on or for regulators to establish meaningful standards.