Revisiting FunnyBirds evaluation framework for prototypical parts networks

How you visualize an AI explanation matters significantly: The FunnyBirds benchmark initially used different visualization methods for different AI models (bounding boxes for ProtoPNet, attribution maps for others), which led to unfair comparisons and potentially misleading conclusions about which models provide better explanations.

Standardized evaluation methods are crucial for trustworthy AI claims: The paper demonstrates that prototypical parts networks like ProtoPNet can only be fairly evaluated when using consistent visualization techniques across all models being compared, highlighting the importance of rigorous benchmarking standards in responsible AI research.

"Explainability" claims require careful scrutiny: While ProtoPNet was promoted as producing more genuine explanations than other methods, this advantage was largely theoretical until systematic evaluation occurred—suggesting that companies and researchers making interpretability claims about their AI systems should provide robust evidence, not just theoretical arguments.

Technical choices in evaluation can hide or reveal model performance: When similarity maps (rather than bounding boxes) were used to visualize ProtoPNet explanations, the metric scores changed substantially, demonstrating that seemingly minor methodological decisions in AI evaluation can significantly alter conclusions about model trustworthiness and interpretability.