Pretraining Curricula Enable Selective Fine-tuning

Imbalanced pretraining curricula create more separable neural circuits: When AI models learn conflicting tasks in sequence (one early, one late) rather than simultaneously, different tasks develop in distinct, isolated parts of the neural network. This disentanglement was demonstrated through activation patching experiments showing that balanced training routes both tasks through shared pathways, while imbalanced training keeps them separate.

Disentangled representations improve the precision of safety fine-tuning: The paper shows that when tasks are learned in imbalanced curricula, subsequent fine-tuning to suppress unwanted behaviors (like refusals) becomes more selective and effective. This matters for AI safety because it means safety interventions can target specific misaligned behaviors without accidentally affecting other model capabilities.

Imbalanced curricula promote in-context learning: Models trained with imbalanced pretraining curricula showed improved ability to learn from context, suggesting that the separation of tasks into distinct circuits enables more flexible, adaptive learning patterns—a desirable property for AI systems.

The findings generalize beyond toy tasks: While initial experiments used simplified copy tasks, the researchers validated their approach on a synthetic language learning task involving rule-consistent and rule-violating data, where imbalanced curricula similarly produced more localized rule representations and more robust rule-following behavior.

Curriculum design is a practical lever for AI alignment: These results suggest that how we structure pretraining—a decision made before fine-tuning—can significantly impact how effectively we can later align AI systems through safety fine-tuning, offering a concrete design principle for responsible AI development.