Evaluating a Methodology for Increasing AI Transparency: A Case Study

User-centered documentation can be practical for developers: The study found that AI developers without formal training in user research methods were able to successfully use a structured methodology to create documentation templates, suggesting that transparency efforts don't require specialized expertise to implement.

One-size-fits-all documentation templates have limitations: The research confirms that generic documentation templates cannot adequately address the diverse needs of different stakeholders (such as clinicians, patients, and regulators in healthcare), supporting the case for customized approaches to AI transparency.

Tailored documentation can still be reusable: Despite creating documentation specifically for their healthcare AI models, the team was able to develop templates that could be applied across multiple different models and use cases, indicating that customization and scalability aren't mutually exclusive.

Methodology requires careful cost-benefit analysis: While the user-centered approach proved usable, the paper notes there are both benefits and costs to implementing this methodology, suggesting organizations need to weigh resource investments against transparency gains when deciding whether to adopt such approaches.

Healthcare AI transparency is achievable but needs refinement: This first real-world evaluation of the methodology in healthcare AI provides evidence that systematic approaches to transparency can work in practice, though the authors identify areas where both the methodology itself and supporting tools could be improved.