eXplainable AI for data driven control: an inverse optimal control approach

Black-box controllers can now be explained through their hidden objectives: The paper demonstrates a method to reverse-engineer what a data-driven controller actually cares about by extracting the implicit cost function it optimizes. Rather than treating controllers as unexplainable "black boxes," this approach reveals whether the system prioritizes tracking accuracy, minimizing energy use, or some combination—making its decision-making transparent.

The method goes beyond simple sensitivity analysis: While existing explanation techniques like LIME show which inputs matter most, this inverse optimal control approach provides a more structured understanding by identifying the actual trade-offs the controller is making (e.g., "this controller weights accuracy 70% and energy efficiency 30%"), offering insights that are more relevant to control system design and oversight.

Explanations are localized and practical: The methodology generates explanations for how a controller behaves in specific operating regions rather than attempting one global explanation for all conditions. This is more realistic since controllers often behave differently under different circumstances, making the explanations more actionable for identifying when and why a system might behave unexpectedly.

Counterintuitive controller behavior becomes understandable: The paper notes that the method "sheds light on otherwise counterintuitive or unexpected phenomena," suggesting this approach can help operators and engineers understand why an AI controller made a puzzling decision—critical for building trust and catching potential problems before deployment.