Explaining RhythmFormer: A Systematic XAI Analysis of Periodic Sparse Attention for Remote Photoplethysmography

Transparency gaps in medical AI: Current heart-rate estimation systems using transformers achieve good accuracy on benchmarks, but their decision-making processes remain largely unexplainable—a critical concern as these systems move toward clinical use where understanding why a prediction was made is essential for patient safety and clinician trust.

Visual explanations can be misleading: The paper reveals that standard visualization methods (heatmaps) used to explain AI decisions in this domain appear plausible but don't actually reflect what the model is doing. For example, attention rollout methods restored 95%+ of connections that the model had explicitly removed, showing that "seeing" an explanation doesn't guarantee it's accurate.

Quantitative validation is necessary: The researchers developed measurable metrics (skin coverage and faithfulness scores) to rigorously test whether explanations are trustworthy, rather than relying on human judgment of heatmaps. This approach found that one method (Beyond Intuition) achieved 0.92 faithfulness versus 0.57 for standard methods—a substantial difference that wouldn't be obvious from visual inspection alone.

Better explanation methods exist but need adoption: The study demonstrates that more sophisticated attribution techniques can provide more reliable explanations of AI decisions in medical contexts, but these require deliberate implementation and validation rather than defaulting to simpler, less trustworthy approaches.

Responsible AI requires ongoing validation: The authors emphasize that validation across diverse datasets and patient populations is still needed, highlighting that responsible deployment of medical AI systems demands continuous testing rather than one-time certification.