ConsistencyAI: A Benchmark to Assess LLMs' Factual Consistency When Responding to Different Demographic Groups

LLMs give different facts to different people: The study found that large language models don't always provide consistent factual information depending on who is asking. When researchers posed identical questions to 19 different LLMs while simulating different demographic personas, the models frequently returned factually inconsistent answers, with consistency scores ranging from 0.79 to 0.91 on a scale where 1.0 would be perfect consistency.

Some models are more reliable than others: Grok-3 demonstrated the highest factual consistency across demographic groups, while several lightweight models ranked significantly lower, suggesting that model choice matters when seeking reliable, unbiased information. This variation indicates that responsible AI deployment should consider which models are most consistent in their factual outputs.

Certain topics trigger inconsistent responses: The benchmark revealed that some subjects are more prone to demographic-dependent inconsistency than others. Job market information was the least consistent topic tested, while information about G7 world leaders was most consistent. Notably, sensitive topics like vaccines and the Israeli-Palestinian conflict showed significant variation depending on the LLM provider, raising concerns about potential bias in how these systems handle controversial subjects.

Independent oversight is necessary: Because this benchmark was developed without involvement from LLM providers, it demonstrates the value of independent evaluation for accountability. The researchers released their code and interactive demo publicly, enabling ongoing scrutiny of whether AI systems treat different demographic groups fairly and factually.