Bidirectional Empowerment of Metamorphic Testing and Large Language Models: A Systematic Survey

Metamorphic testing addresses a critical gap in AI quality assurance: Traditional software testing relies on knowing exact correct outputs, but LLMs produce probabilistic, variable responses. Metamorphic testing solves this "oracle problem" by checking logical relationships between multiple test runs instead—for example, verifying that rephrasing a question shouldn't dramatically change an answer's accuracy.

LLMs can automate the tedious work of creating tests: While metamorphic testing is promising for LLMs, it's traditionally labor-intensive to set up. The survey identifies that LLMs themselves can now help automate key testing phases, including discovering what relationships to test, generating test inputs, and writing test code—creating a mutually beneficial cycle.

This approach helps catch serious AI failures across multiple domains: The survey identifies metamorphic testing successfully detecting problems in LLMs including hallucinations (false information), fairness issues, robustness failures, code generation errors, and problems in specialized systems like retrieval-augmented generation and autonomous agents—suggesting broad applicability for responsible AI.

The field is rapidly evolving with structured frameworks emerging: By systematically reviewing 93 studies, this survey provides the first comprehensive taxonomy of how metamorphic testing and LLMs work together, establishing a foundation for future development and suggesting this is becoming a standard approach rather than an experimental technique.