Prompt-Reverse Inconsistency: LLM Self-Inconsistency Beyond Generative Randomness and Prompt Paraphrasing

LLMs struggle with basic logical consistency: The research identifies a new type of inconsistency called Prompt-Reverse Inconsistency (PRIN), where language models give contradictory answers when asked "Which answers are correct?" versus "Which answers are incorrect?" about the same content. This suggests LLMs fail to adhere to fundamental logical rules, even when the questions are logically inverse.

Using LLMs as judges is unreliable: Since PRIN undermines LLM credibility when evaluating answers, organizations relying on language models to assess or judge content quality, correctness, or safety should be cautious. This inconsistency could lead to flawed automated decision-making in critical applications.

Inconsistency goes beyond randomness and wording: While previous research attributed LLM inconsistency to random variation or prompt paraphrasing, this study reveals a deeper logical problem. PRIN represents a distinct category of self-contradiction that persists even when controlling for these other factors, indicating a fundamental limitation in how LLMs process logical relationships.

The problem is widespread across different models: The research examines PRIN "across different LLMs," suggesting this is not an isolated issue with one model but rather a systemic challenge affecting multiple language models, making it a broader concern for the AI industry.

Mitigation strategies exist but need development: The authors investigate "methods to mitigate" PRIN, indicating that while the problem is serious, there are potential solutions worth exploring to improve LLM reliability and trustworthiness in real-world applications.