FaceX: Understanding Face Attribute Classifiers through Summary Model Explanations

FaceX addresses a critical gap in AI explainability for facial analysis: While existing explanation methods show why a model made a decision for individual images, they require labor-intensive manual inspection of thousands of images to understand overall model behavior. FaceX provides "summary model explanations" that reveal patterns across entire datasets, making it feasible to assess fairness issues systematically rather than case-by-case.

The method identifies which facial features drive model decisions across 19 distinct regions: FaceX visualizes how much different parts of the face (hair, ears, skin, etc.) influence a model's attribute classifications. This regional breakdown helps identify whether models are relying on problematic features—for example, if a model disproportionately uses skin tone to make predictions, this bias becomes immediately visible.

FaceX was tested on multiple real-world facial datasets and bias scenarios: The evaluation included benchmarks like CelebA and FairFace, and tested both models with and without intentional biases and bias mitigation efforts. This comprehensive testing demonstrates the method's practical applicability for auditing facial recognition systems used in real applications.

The tool moves beyond abstract explanations to show concrete examples: Beyond visualizing which regions matter, FaceX displays actual image patches that most strongly influence the model's decisions for each facial region, making the explanations more concrete and actionable for non-technical stakeholders reviewing fairness concerns.