Measuring the Effect of Background on Classification and Feature Importance in Deep Learning for AV Perception

Background bias significantly impacts model decisions: The paper demonstrates that deep learning models used in autonomous vehicle perception can rely heavily on background features rather than the actual object of interest (traffic signs), which represents a genuine risk for real-world deployment where backgrounds may differ from training data.

Current explainability methods lack quantitative validation: Popular XAI techniques like SHAP and GradCAM show which image regions influence classification, but the paper reveals these methods themselves lack rigorous quantitative testing—making it difficult to definitively determine whether a model is truly learning the right features or just appearing to do so.

Synthetic data is essential for testing AI robustness: By creating controlled synthetic datasets that systematically vary background correlation and camera angles independently, the researchers show that synthetic environments allow researchers to isolate and measure specific sources of bias that are nearly impossible to control in real-world data.

Correlations in training data are ambiguous: The paper highlights a critical challenge: distinguishing between "spurious" correlations (misleading patterns the model shouldn't learn) and "legitimate" ones is often debatable in real data, making it harder to know whether a model's reliance on background is a flaw or acceptable behavior.

Camera variation matters for generalization: The systematic testing reveals that different levels of camera variation in training data influence how much models depend on background features, suggesting that diverse training conditions are important for building more robust autonomous vehicle perception systems.