RAIN: Your Language Models Can Align Themselves without Finetuning

Models can self-correct without additional training data: RAIN demonstrates that pre-trained language models can align themselves with human preferences through self-evaluation and "rewinding" during inference alone, eliminating the need for costly human preference data collection or model fine-tuning—a significant practical advantage for deploying safer AI systems.

Dramatic safety improvements are achievable at inference time: On the HH dataset, RAIN increased harmlessness rates from 82% to 97% for LLaMA 30B while preserving helpfulness, showing that substantial safety gains don't necessarily require retraining models but can instead be achieved through smarter inference methods.

The approach works by having models evaluate themselves: The method relies on LLMs assessing the quality and safety of their own responses, then using those self-evaluations to guide which outputs to keep or regenerate—essentially enabling models to catch and correct their own problematic outputs in real-time.

Even well-aligned models can improve further: RAIN improved truthfulness by 5% on TruthfulQA for LLaMA-2-chat 13B, which was already considered well-aligned, suggesting the technique could provide incremental safety gains across different model types and training stages.

No computational overhead from training or gradients: Unlike traditional alignment methods, RAIN requires no gradient computation, parameter updates, or additional training, making it potentially more efficient and easier to implement as a post-deployment safety layer for existing models.