The logic of NTQR evaluations of noisy AI agents: Complete postulates and logically consistent error correlations

The "evaluation dilemma" is a fundamental challenge in AI safety: The paper formalizes Plato's ancient concern about evaluating experts when you cannot trust the evaluators themselves. In unsupervised settings where we lack labeled data, we face an endless validation chain—how do we know our evaluation algorithm is correct if we have no ground truth to verify it against?

Mathematical consistency checks can validate evaluation algorithms without labeled data: Rather than relying on external validation, the paper proposes using "complete postulates"—algebraic functions that can prove or disprove whether a grading algorithm is logically consistent. This offers a way to detect flawed evaluation methods purely through mathematical analysis of observed responses.

Error correlations between AI agents matter for fair evaluation: The research shows that when evaluating multiple AI agents, their errors are often correlated rather than independent. The paper demonstrates how to calculate these error correlations algebraically, which is important because ignoring correlations could lead to systematically biased assessments of which agents perform best.

Existing ML techniques already use these principles without recognizing them: The paper identifies that "agreement equations" previously developed in machine learning literature are actually instances of these formal postulates, suggesting that responsible AI evaluation practices may already exist in the field but lack unified theoretical grounding.