Do Machines Fail Like Humans? A Human-Centred Out-of-Distribution Spectrum for Mapping Error Alignment

AI systems can match human accuracy on standard tasks but may use fundamentally different decision-making strategies, meaning that similar performance scores don't guarantee similar reasoning—a critical distinction for trustworthy AI systems.

The paper introduces a "human-centered" framework that measures AI difficulty based on actual human perception rather than arbitrary technical parameters, allowing for more meaningful comparisons between how humans and machines fail on challenging tasks.

Vision-language models showed the most consistent alignment with human error patterns across different difficulty levels, suggesting this type of AI architecture may be more trustworthy for applications requiring human-like decision-making.

AI systems show "regime-dependent" performance patterns, meaning their alignment with human thinking changes dramatically depending on task difficulty—they may think like humans on easy tasks but diverge significantly as challenges increase, which has implications for deploying AI in real-world scenarios.

Existing methods for testing AI reliability are flawed because they use arbitrary distortion parameters disconnected from human perception, highlighting the need for more human-centered evaluation approaches to properly assess whether AI systems are truly trustworthy.