Explaining Facial Expression Recognition

Multimodal explanations improve understanding: The study found that combining visual and textual explanations of facial expressions—showing both what facial movements the AI detected and describing them in words—led to better user comprehension compared to other explainable AI methods.

Text-only explanations can be effective: Interestingly, textual-only explanations using Facial Action Units (the specific muscle movements that indicate emotions) were nearly as effective as combined visual-textual approaches, suggesting that detailed written descriptions alone can meaningfully help users understand AI decisions.

Explanations build appropriate trust: All versions of the FAU-based explanation method improved users' trust in the facial recognition system, addressing a key challenge in deploying emotion-recognition AI in real-world applications where users need confidence in the technology.

Transparency matters for real-world deployment: The research highlights that simply having accurate facial expression recognition isn't enough—users need to understand how and why the system reached its conclusions, which is essential before deploying such systems in sensitive contexts like healthcare, education, or customer service.

Facial Action Units provide interpretable explanations: By grounding explanations in specific, observable facial movements rather than abstract AI outputs, the method makes emotion detection more transparent and verifiable to end users.