Creative Uses of AI Systems and their Explanations: A Case Study from Insurance

AI systems are often used in ways their designers never intended. The study found that insurance professionals creatively adapted AI tools and explanations for purposes beyond their original design, highlighting a gap between how developers expect systems to be used and how they actually function in real-world workflows.

Different stakeholders need different explanations. The research involved participants with varying professional roles and backgrounds, suggesting that a one-size-fits-all approach to AI explanations is insufficient—insurance agents, managers, and other professionals require tailored information to effectively use AI systems responsibly.

Current explainability methods may create unforeseen risks. The paper warns that existing approaches to explaining AI decisions "fall short at intercepting and managing unexpected user behavior," meaning that explanations designed to increase transparency could inadvertently enable misuse or workarounds if not carefully designed for specific contexts.

Human-centered design is essential but incomplete. While the authors acknowledge that recent work has emphasized human-centered perspectives in AI design, this case study demonstrates that even well-intentioned approaches can miss how real users will actually interact with systems across different professional contexts.

Responsible AI requires ongoing monitoring and adaptation. The study recommends that organizations designing AI systems should actively manage the risks of unexpected uses while capturing their potential benefits—suggesting that responsible AI is not a one-time design decision but an ongoing process of observation and refinement.