When Superpixels Fail on Documents: A Study of Segmentation for LIME Explanations

Explanation methods require domain-specific design choices: The paper demonstrates that LIME, a widely-used tool for understanding AI decisions, produces unreliable explanations when applied to documents using generic image segmentation. This suggests that practitioners cannot simply apply explanation tools "off-the-shelf" without considering the specific type of data being analyzed.

Standard segmentation methods can hide AI bias: When researchers used document-aware segmentation (based on text regions and layout), they discovered that the AI classifier was relying on document identification codes—a known bias in the dataset that standard image-based segmentation had previously obscured. This finding highlights how explanation method choices can either reveal or conceal problematic AI behavior.

Segmentation is not a neutral preprocessing step: The paper shows that segmentation "strongly affects explanation consistency, correctness, and local fidelity," meaning the choice of how to break down an image fundamentally changes what the explanation reveals about the model's decision-making. This challenges the common practice of treating segmentation as a minor technical detail rather than a critical component of the explanation process.

Document-aware approaches improve explanation reliability: Using segmentation methods aligned with document structure (OCR bounding boxes and grids) produced more stable explanations that required fewer computational steps to converge, suggesting that domain knowledge should inform how we design tools to inspect AI systems.