Towards Trustworthy and Explainable AI for Perception Models: From Concept to Prototype Vehicle Deployment

Explainability is achievable in real-world autonomous driving: The researchers successfully deployed a perception system in an actual prototype vehicle that generates visual explanations (saliency maps) showing which parts of images the AI uses to make driving decisions, demonstrating that XAI isn't just theoretical but can work in practice.

Uncertainty quantification matters for safety: The system includes calibrated uncertainty estimates—essentially confidence scores—that indicate when the AI is less sure about its decisions. This allows human operators and safety systems to know when to be more cautious, addressing a critical gap in current autonomous driving systems.

Attention mechanisms can provide faithful explanations: By extracting explanations from the transformer model's attention mechanism and validating them through perturbation tests (removing image regions to verify the explanations are accurate), the researchers showed that deep learning models can provide trustworthy insights into their decision-making rather than just "black box" outputs.

Multiple safeguards work together: The approach combines three elements—robustness training, explainability, and uncertainty calibration—suggesting that trustworthy AI requires layered protections rather than relying on any single technique to ensure safe autonomous systems.

Real-time monitoring is feasible: The deployed XAI interface visualizes the model's reasoning, uncertainty state, and confidence in real time, making it practical for human operators to monitor and oversee autonomous vehicle perception systems during operation.