Iterative Deployment Improves Planning Skills in LLMs

Iterative deployment of LLMs can unintentionally function as reinforcement learning: The paper demonstrates that repeatedly deploying models and fine-tuning them on user-curated data from previous versions effectively implements RL training "in the outer-loop," meaning this optimization happens through deployment cycles rather than explicit training—a mechanism that may occur without developers fully intending it.

An implicit, undefined reward function emerges from user curation: Because the "reward signal" comes from which examples users choose to curate rather than from explicitly programmed objectives, "the reward function entailed by repeated deployment is not defined explicitly, and could have unexpected implications to the properties of future model deployments"—raising concerns about what behaviors are actually being optimized for.

Planning capabilities improve substantially but may generalize unpredictably: The models showed "substantial improvements in planning skills" with "emergent generalization by discovering much longer plans than the initial models," but the paper suggests this emergent behavior wasn't necessarily anticipated, highlighting risks when AI systems develop capabilities through iterative deployment rather than controlled training.

Data curation can be an alternative to explicit reward design, but with hidden tradeoffs: The mechanism offers "an alternative training regime to explicit RL, relying on data curation rather than explicit rewards," which could be simpler to implement but obscures what objectives are actually being reinforced—making it harder to predict or control model behavior.