GP-VM$\times$SMA: Benchmarking General-Purpose Vision Models and Specialized Architectures for 2D Medical Image Segmentation

General-purpose vision models perform comparably to specialized medical models: When tested across three different 2D medical imaging datasets using standardized training protocols, general-purpose vision models (originally designed for natural images) achieved segmentation performance on par with architectures specifically built for medical imaging, challenging the assumption that domain-specific design is necessary for medical tasks.

Specialized medical architectures may not provide systematic advantages: The study found no consistent evidence that the eleven specialized medical segmentation architectures tested provided systematic performance benefits over modern general-purpose models, suggesting that the substantial effort invested in creating medical-specific designs may not always be justified.

General-purpose models can identify clinically relevant structures without explicit medical training: Explainability analysis using Grad-CAM visualizations showed that general-purpose vision models were capable of identifying and focusing on clinically relevant anatomical structures, even though they were not explicitly designed with medical domain knowledge, indicating they can learn meaningful medical features from data alone.

Model choice may depend on practical considerations beyond accuracy: Since general-purpose and specialized models performed similarly on segmentation metrics, decisions about which model to deploy could be based on other factors such as computational efficiency, ease of implementation, availability of pre-trained weights, or organizational infrastructure rather than assumed performance superiority.

Standardized evaluation protocols reveal important insights: The use of unified training procedures, statistical testing via linear mixed-effects models, and multiple diverse datasets was essential to fairly compare these different approaches, highlighting the importance of rigorous benchmarking methodology in AI research.