Using Sentence Embeddings and Semantic Similarity for Seeking Consensus when Assessing Trustworthy AI

Trustworthy AI assessment requires bridging disciplinary language gaps: The paper identifies a fundamental challenge in evaluating AI trustworthiness—experts from different fields (e.g., computer science, ethics, medicine) often use different terminology to describe the same problems, creating communication barriers that can hinder comprehensive assessment.

Semantic similarity tools can help experts find common ground: The researchers demonstrated that natural language processing techniques, specifically sentence embeddings and semantic textual similarity, can automatically identify when experts are discussing identical issues despite using different words or specialized jargon, supporting more effective interdisciplinary collaboration.

Terminology differences pose real risks to AI evaluation: The paper highlights that without tools to identify semantic overlap, experts might fail to recognize they're addressing the same trustworthiness concern, potentially leading to incomplete or duplicative assessments of AI systems—particularly critical in high-stakes domains like healthcare.

Practical application in healthcare AI: The research was tested on a real-world case of assessing trustworthiness in a healthcare AI system, suggesting these semantic matching techniques have concrete applicability to evaluating AI systems in sensitive domains where trustworthiness is paramount.