Quantifying Fidelity: A Decisive Feature Approach to Comparing Synthetic and Real Imagery

Visual realism alone is insufficient for safe autonomous vehicle testing: The paper demonstrates that synthetic data looking realistic to humans does not guarantee that AI systems will make decisions the same way in both simulated and real environments—a critical gap for safety assurance that pixel-level quality metrics miss.

What matters is consistent decision-making logic, not just matching outputs: The research introduces "Decisive Feature Fidelity" (DFF), which measures whether autonomous vehicles base their decisions on the same evidence in both real and synthetic environments. This mechanism-level agreement is more important than whether final outputs numerically match.

Current fidelity measures overlook important safety problems: Testing on over 2,100 real-world and simulated image pairs revealed that conventional metrics failed to detect discrepancies that DFF identified, meaning current validation approaches may give false confidence in simulator safety.

AI explainability tools can improve simulator calibration: By using explainable-AI methods to identify which features drive an autonomous vehicle's decisions, researchers developed a calibration approach that improved fidelity across multiple types of AI systems without degrading performance on standard metrics.

Simulator validation requires system-specific assessment: Rather than a one-size-fits-all fidelity score, the DFF approach is tailored to each specific autonomous vehicle system, recognizing that different AI models may rely on different decision evidence and therefore need customized validation.