Infra-Bayesian Reinforcement Learning Agents Outperform Classical RL For Worst-Case Robustness

Classical RL assumes environments won't change based on agent behavior, but this breaks down in real-world AI safety scenarios where the agent interacts with predictors, humans, other AI systems, and institutions that actively anticipate and respond to the agent's actions.

Standard Bayesian methods can produce "confidently wrong" decisions when their model assumptions don't match reality, leading to unreliable outcomes and unbounded regret—a critical concern for AI systems deployed in complex, partially understood environments.

Infra-Bayesianism distinguishes between two types of uncertainty: ordinary probabilistic uncertainty (where reasonable assumptions can be made) and Knightian uncertainty (where no solid grounds exist for assumptions), addressing a gap that classical methods ignore.

The proposed infra-Bayesian RL agent uses worst-case reasoning rather than averaging over possibilities, maintaining a set of imprecise hypotheses and selecting actions that perform best in the worst plausible scenario—a more conservative approach to decision-making under deep uncertainty.

This represents the first working implementation of infra-Bayesian RL for practical problems, demonstrating that worst-case optimization can outperform classical RL when environments contain fundamental uncertainties that violate standard assumptions.