Towards Optimising EEG Decoding using Post-hoc Explanations and Domain Knowledge

Accuracy alone is insufficient for trustworthy AI systems: A model achieving 72.60% accuracy performed nearly identically to one using only neurophysiologically relevant data (70.85% accuracy), yet the two models relied on completely different features. This demonstrates that high performance metrics can mask whether an AI system is actually learning the "right" patterns for the right reasons.

Explainability tools reveal hidden problems: Using post-hoc explanation techniques (GradCAM), researchers discovered that a high-performing model was relying on features that contradicted established neuroscience knowledge about motor imagery. Without these explanations, this problematic model would have been deployed based on accuracy scores alone.

Domain expertise is essential for AI validation: Integrating neurophysiological knowledge with AI explainability techniques proved critical for validating whether BCI models were functioning correctly. This suggests that responsible AI deployment requires collaboration between AI developers and subject matter experts who can assess whether models are learning scientifically sound patterns.

Performance metrics can create false confidence: The paper warns against "exclusively relying on performance metrics when selecting models for dependable and transparent BCIs." This applies broadly to high-stakes applications like medical devices and brain-computer interfaces where understanding why a model works matters as much as whether it works.

Transparency is especially critical for medical and assistive technologies: BCIs directly interface with the human brain and serve people with disabilities. The paper emphasizes that model interpretability and neurophysiological validation are not optional features but essential requirements for systems that directly impact human health and autonomy.