Space-scale Exploration of the Poor Reliability of Deep Learning Models: the Case of the Remote Sensing of Rooftop Photovoltaic Systems

Deep learning models for detecting rooftop solar panels are currently unreliable for real-world use by government agencies and grid operators because they perform poorly when faced with different imaging conditions than their training data—a problem called "distribution shift."

The researchers identified specific sources of unreliability by systematically testing how factors like different camera angles, lighting conditions, image resolution, and geographic locations affect the model's ability to accurately identify solar panels, demonstrating that no single factor alone explains the poor performance.

Using explainable AI tools revealed that models make decisions based on image features at different scales, and distribution shifts disproportionately affect how the model processes certain scales of visual information, providing insight into why models fail rather than just that they fail.

A targeted data augmentation technique improved model robustness, suggesting that training models with more diverse and varied imagery of solar panels can help them perform more consistently across different real-world imaging conditions—though the paper indicates this remains an ongoing challenge.

This research highlights a critical gap between AI capability and deployment readiness: while deep learning shows promise for monitoring distributed solar energy (important for grid stability and decarbonization), the technology cannot yet reliably support policy decisions without further development to handle real-world variability.