AgentSHAP: Interpreting LLM Agent Tool Importance with Monte Carlo Shapley Value Estimation

LLM agents using external tools are increasingly common, but we lack transparency about which tools actually drive their decisions. AgentSHAP addresses this gap by providing the first explainability method specifically designed to measure tool importance in agent systems, filling a previously unaddressed need in responsible AI.

The framework works with any LLM without requiring access to internal model weights or code, making it practical for real-world deployment across different AI systems (GPT, Claude, Llama, etc.). This "black box" approach means organizations can audit tool usage even when using proprietary models.

AgentSHAP uses game theory-based Shapley values to fairly attribute importance, testing how agents respond when different tool combinations are available. This principled mathematical approach prevents arbitrary or biased importance rankings that could mislead users about which tools actually matter.

The method reduces computational cost from exponential O(2^n) complexity to practical levels through Monte Carlo sampling, making tool attribution feasible for real agents with many tools rather than remaining a theoretical exercise.

Experiments demonstrate AgentSHAP consistently identifies which tools are genuinely relevant versus irrelevant, suggesting the method could help organizations detect when agents are relying on inappropriate tools or when tool configurations are inefficient—important for maintaining control and trust in AI systems.