Position: Multi-Agent Algorithmic Care Systems Demand Contestability for Trustworthy AI

Explainability alone is insufficient for trustworthy AI in healthcare: The paper argues that while explainability helps users understand AI decisions, it does not enable care partners to actually challenge or correct system outputs—a critical gap in multi-agent healthcare systems where multiple specialized agents collaborate on decisions.

Contestability requires three key components: Trustworthy multi-agent care systems need transparency (understanding how decisions are made), structured opportunities for intervention (chances to challenge decisions), and mechanisms for review, correction, or override—going beyond current explainability-focused approaches.

Multi-agent systems create unique accountability challenges: Because these systems act as collective decision-makers rather than single entities, they raise distinct problems for trust and human oversight that existing trustworthy AI frameworks don't adequately address, particularly in high-stakes healthcare contexts.

Human agency and clinical responsibility must be preserved: The paper proposes a human-in-the-loop framework using structured argumentation and role-based contestation to ensure that healthcare professionals retain meaningful control and accountability rather than deferring to algorithmic recommendations.

Current AI governance approaches are inadequate for collaborative systems: The paper identifies key limitations in both multi-agent systems research and explainable AI research, suggesting that new design requirements—specifically contestability—are necessary to build trustworthy algorithmic care systems.