Unveiling the Human-like Similarities of Automatic Facial Expression Recognition: An Empirical Exploration through Explainable AI

AI models don't process faces the same way humans do: Despite achieving high accuracy rates, the study found low alignment between how deep learning models and humans identify important facial regions for emotions, with average Intersection over Union scores of only 0.27 out of 1.0, suggesting these systems may reach correct answers through different reasoning than human perception.

Pre-training matters for human-like behavior: Models that were pre-trained on general image recognition tasks showed more similarity to human facial processing patterns than models trained from scratch, indicating that exposure to broader visual concepts helps AI systems develop more human-aligned decision-making processes.

Different facial features matter for different emotions: The research identified that eyes and nose regions are particularly influential for certain expressions, while the mouth region consistently matters across all emotions—findings that align with established human emotion recognition science and suggest AI systems have learned some genuine patterns about facial expressions.

Explainability reveals a transparency gap: By using visualization techniques to show which facial regions AI models focus on, the study demonstrates that explainable AI methods are essential for understanding whether automated systems are making decisions for the "right reasons," even when their accuracy appears acceptable.

One-size-fits-all AI approaches may be problematic: The significant variation in how different neural network architectures process facial expressions (best performer at 0.33 IoU vs. worst at 0.21) suggests that deploying facial recognition systems without understanding their underlying decision-making could lead to inconsistent or unreliable real-world applications.