What Did I Learn? Operational Competence Assessment for AI-Based Trajectory Planners

Training data gaps create real safety risks: The paper demonstrates that AI systems for autonomous vehicles perform only as well as their training data allows. By analyzing the NuPlan dataset, researchers found that certain driving scenarios had insufficient coverage, meaning vehicles trained on this data may not be adequately prepared for those real-world situations.

We need better ways to audit what AI systems have "learned": The researchers developed a method using knowledge graphs to map out exactly what driving situations are represented in training datasets and at what complexity levels. This addresses a critical gap in AI transparency—currently, it's difficult to know what scenarios an autonomous vehicle has actually been trained to handle.

Complexity matters as much as quantity: The paper introduces an important principle: rare or complex driving situations require more extensive training examples than simple ones. A vehicle might see 1,000 highway merges but still be unprepared if it only encountered 10 examples of complex multi-vehicle interactions in poor weather—a finding that challenges simplistic "more data is better" approaches.

Explainability and safety go hand-in-hand: By describing datasets in human-understandable terms (rather than just raw statistics), this method makes it possible for safety engineers, regulators, and the public to understand what an autonomous vehicle is and isn't prepared for—essential information for trustworthy deployment.

This is a practical step toward responsible AI deployment: The approach provides a concrete tool for monitoring whether machine learning models are competent for real-world use before they're deployed on public roads, addressing a key requirement for safe autonomous driving systems.