COCKATIEL: COntinuous Concept ranKed ATtribution with Interpretable ELements for explaining neural net classifiers on NLP tasks

Existing explanation methods for AI language models are unreliable: The paper demonstrates that commonly used techniques like attention maps and attribution methods fail to accurately explain how transformer-based language models make decisions, highlighting a significant gap in current AI transparency practices.

COCKATIEL offers a more trustworthy alternative: The new method discovers the actual concepts that AI models use to classify text (such as sentiment) by analyzing the model's final layer, and then measures how important each concept is to the final decision—without requiring retraining or sacrificing model accuracy.

Human-AI concept alignment matters for trust: Testing on sentiment analysis tasks, COCKATIEL successfully identified concepts that matched human understanding better than previous methods, suggesting that explanations aligned with human reasoning may be more reliable for building trust in AI systems.

Explanations can be verified for accuracy: The researchers validated their explanations using "fidelity metrics"—objective measures that confirm whether the identified concepts actually reflect what the model is doing—addressing concerns that some explanations might sound plausible but be misleading.

Better explanations don't require starting from scratch: Because COCKATIEL works on already-trained models without modification, organizations can improve the explainability of their existing AI systems without the cost and complexity of retraining, making responsible AI practices more practical to implement.