RefereeBench: Are Video MLLMs Ready to be Multi-Sport Referees

Current AI systems are not ready for real-world decision-making in specialized domains: Even the best video AI models achieve only around 60% accuracy at making referee calls in sports, falling far short of the reliability needed for actual officiating. This suggests that deploying AI for high-stakes decisions in specialized fields requires significant additional development.

AI struggles with applying rules consistently, not just understanding what's happening: While the models tested could often identify when incidents occurred and which players were involved, they frequently failed at the core task—correctly applying the actual rules of the game. This highlights a critical gap between general video understanding and domain-specific reasoning that responsible AI deployment must address.

AI systems have a bias toward over-enforcement: The models tested frequently made false calls on normal plays, suggesting they tend to err on the side of caution in ways that could unfairly penalize athletes. This type of systematic bias is important to identify and correct before deploying AI in any decision-making role affecting people.

Benchmarking specialized AI capabilities is essential for responsible deployment: RefereeBench's creation of a large, human-verified dataset (925 videos across 11 sports) demonstrates the importance of rigorous, domain-specific evaluation before claiming AI systems are ready for real-world use—a model that should apply to other high-stakes applications.

Trustworthy AI-assisted decision-making requires integrating domain expertise: The paper emphasizes that future systems need better integration of actual rule knowledge alongside visual understanding, suggesting that responsible AI development in specialized fields must involve deep collaboration with domain experts, not just general-purpose AI advancement.