Highlight All the Phrases: Enhancing LLM Transparency through Visual Factuality Indicators

Visual highlighting of individual phrases improves user trust: The study found that color-coding all phrases in an LLM response based on factuality scores led to higher trust ratings compared to other design approaches, suggesting that granular, transparent feedback helps users feel more confident in AI outputs.

Phrase-level factuality indicators make it easier for users to verify accuracy: Participants reported significantly greater ease in validating whether an LLM's response was accurate when phrases were individually color-coded, compared to responses with no visual indicators—a practical benefit for users who want to fact-check AI-generated content.

Users prefer comprehensive transparency over minimal design: Among the design strategies tested with 208 participants, the approach that highlighted all phrases (rather than only uncertain or problematic content) was both most preferred and most trusted, indicating that users value complete visibility into the model's confidence levels.

Better communication of AI uncertainty can calibrate user expectations: By making factuality scores visible through design, the research addresses a critical gap between technical AI safety advances and user understanding, helping prevent both over-reliance on AI systems and unwarranted dismissal of accurate information.

Design choices matter for responsible AI deployment: The findings provide actionable guidelines for developers and designers, demonstrating that how uncertainty information is presented to users significantly impacts their ability to use AI systems responsibly and critically.