Explaining Text Similarity in Transformer Models

Transformer models used for text similarity lack transparency: The paper identifies that similarity models built on Transformer foundation models—widely used in information retrieval and search applications—operate as "black boxes" whose decision-making processes remain largely unexplained, raising concerns about their reliability in critical applications.

New explanation techniques can reveal how AI judges text similarity: Using a method called BiLRP (layer-wise relevance propagation), researchers successfully identified which specific feature interactions drive similarity judgments in these models, demonstrating that explainable AI methods can make opaque systems more interpretable.

AI similarity models show consistent patterns across different languages and domains: The study validated explanations across three real-world use cases—grammatical analysis, multilingual semantics, and biomedical text retrieval—suggesting that understanding one application may provide insights applicable to others, which is valuable for responsible deployment across sectors.

Explainability enables detection of potential model biases: By examining which features drive similarity decisions at the corpus level, organizations can identify whether models rely on problematic patterns (such as grammatical shortcuts rather than semantic meaning) before deploying them in high-stakes contexts like medical information retrieval.

Responsible AI requires moving beyond accuracy metrics: The paper demonstrates that understanding how models make predictions—not just whether they're accurate—is essential for trustworthy AI systems, particularly in unsupervised applications where outcomes directly affect users' access to information.