SHAP-Weighted Cross-Modal Expert Fusion for Emotion and Sentiment Recognition: Evidence and Limits

Explainability can improve AI performance: The paper demonstrates that using SHAP (a popular explainability method) to guide how different data sources are combined actually improves emotion recognition accuracy, suggesting that understanding why an AI makes decisions can lead to better decisions overall.

How you measure importance matters: The researchers found that different mathematical approaches to calculating feature importance from SHAP values can significantly bias results—specifically, they showed that certain methods unfairly suppressed high-dimensional data sources (like video combined with audio), highlighting how technical implementation choices in explainability tools can have real consequences.

Combining multiple data sources requires careful design: The study compared three fusion strategies (early, late, and adaptive), finding that simply averaging predictions from separate models (late fusion) performed worst, while the explainability-guided approach nearly matched the best-performing method, suggesting that thoughtful integration of multimodal information is crucial for responsible AI systems.

Cross-modal interactions matter for emotion recognition: Adding experts that combine multiple data types (video, audio, text together) provided the largest performance gains, indicating that AI systems should be designed to capture how different information sources interact rather than treating them independently.

Rigorous statistical testing is essential: The authors used McNemar testing to verify their results were statistically significant rather than due to chance, demonstrating best practices for validating AI improvements in responsible research.