Beyond Accuracy: An Explainability-Driven Analysis of Harmful Content Detection

High accuracy metrics mask real-world failures: Despite achieving 0.94 accuracy and 0.93 AUC, the model showed systematic failure modes invisible in aggregate performance numbers, particularly with borderline cases, contextual toxicity, and politically sensitive content—demonstrating that accuracy alone is insufficient for evaluating content moderation systems.

Different explanation methods reveal different model behaviors: The two explainability techniques analyzed (Shapley Additive Explanations and Integrated Gradients) produced diverging outputs, with one focusing on explicit words and the other on diffuse context, suggesting that choosing an explanation method significantly impacts what we understand about why a model makes decisions.

Models struggle with context-dependent harmful content: The analysis identified recurring failure modes including indirect toxicity, over-reliance on specific words, and mishandling of political discourse, indicating that current neural models cannot reliably distinguish between harmful and acceptable uses of similar language depending on context.

Explainability enables better human oversight: By exposing model limitations through explanation methods, the research suggests that explainable AI can support "human-in-the-loop moderation," allowing human moderators to understand and correct model errors rather than blindly trusting automated decisions.

Responsible AI requires transparency beyond performance metrics: The paper demonstrates that responsible deployment of content moderation systems requires examining why models make predictions through explainability analysis, not just whether they achieve high accuracy.