BioDSA-1K: Benchmarking Data Science Agents for Biomedical Research

AI agents struggle with real-world biomedical research tasks: The benchmark reveals that validating scientific hypotheses remains "difficult for artificial intelligence agents due to the complexity of real-world data analysis and evidence interpretation," suggesting current AI systems are not yet ready to independently conduct biomedical research without human oversight.

The benchmark includes an important "unknown" category: BioDSA-1K incorporates "non-verifiable hypotheses" where available data are insufficient to support or refute claims, reflecting real-world scenarios where AI must recognize the limits of what can be concluded—a critical capability for responsible AI that avoids false certainty.

Evaluation goes beyond simple accuracy: The benchmark assesses AI agents across four dimensions including "alignment between evidence and conclusion" and "correctness of the reasoning process," not just whether they reach the right answer. This means responsible AI evaluation should examine how agents reach conclusions, not just what conclusions they reach.

The benchmark is grounded in authentic research practices: With 1,029 tasks curated from over 300 published biomedical studies, BioDSA-1K reflects "the structure and reasoning found in authentic research workflows," making it a realistic test for whether AI can be trusted in actual scientific contexts rather than simplified scenarios.

Transparency in AI reasoning is measurable: The inclusion of "executability of the AI-generated analysis code" as an evaluation criterion emphasizes that responsible AI systems should produce interpretable, verifiable work products that humans can audit and understand.