Do Segmentation Models Understand Vascular Structure? A Blob-Based XAI Framework

Current segmentation models may not understand vessel anatomy the way clinicians do: The study found that deep learning models make decisions based primarily on localized visual cues rather than understanding global anatomical structures like how vessels connect or branch. This suggests models are pattern-matching rather than reasoning about vascular structure, which could be problematic for clinical trust.

Explainability tools reveal what AI models actually "see": By developing a specialized analysis framework combining saliency maps and blob detection, researchers demonstrated that we can now visualize which parts of medical images influence model decisions. This type of structured explainability is essential for determining whether AI systems are making decisions for the "right reasons" from a medical perspective.

Model confidence may not reflect anatomical understanding: The analysis showed that model attribution features had little correlation with important vessel properties like thickness, tubularity, or connectivity. This means a model could achieve high accuracy scores while fundamentally misunderstanding vascular anatomy—a critical distinction for clinical safety.

Black-box medical AI poses adoption barriers: The paper emphasizes that impressive performance metrics alone are insufficient for clinical adoption. Healthcare professionals need to understand why a model makes segmentation decisions, not just that it makes them accurately, making explainability tools a prerequisite for responsible medical AI deployment.