REX: Reasoning-aware and Grounded Explanation

AI systems need to explain their reasoning, not just get answers right: The paper found that most visual reasoning AI focuses on accuracy while neglecting to explain why decisions are made, allowing systems to exploit spurious biases rather than perform genuine reasoning. REX addresses this by creating explanations that show the step-by-step reasoning process.

Connecting images and text is crucial for trustworthy AI: The research demonstrates that effective explanations require "tightly coupling" information from both visual and textual data—matching specific words to regions in images. This cross-modal grounding significantly improved both interpretability and reasoning performance compared to systems that treat these modalities separately.

A new dataset enables better AI transparency: The authors created a dataset of over 1 million multi-modal explanations using functional programs that sequentially execute reasoning steps. This resource allows AI systems to learn how to provide grounded, interpretable explanations rather than just predictions.

Explainability and accuracy can reinforce each other: Rather than being trade-offs, the paper shows that when AI systems are trained to explain their reasoning through multi-modal grounding, their overall reasoning performance actually improves—suggesting that interpretability and effectiveness are complementary goals for trustworthy AI.