SafetyTrustworthinessAccountabilitycs.CL
Aurora-M: Open Source Continual Pre-training for Multilingual Language and Code
Taishi Nakamura, Mayank Mishra, Simone Tedeschi, Yekun Chai, Jason T Stillerman, Felix Friedrich, Prateek Yadav, Tanmay Laud, Vu Minh Chien, Terry Yue Zhuo, Diganta Misra, Ben Bogin, Xuan-Son Vu, Marzena Karpinska, Arnav Varma Dantuluri, Wojciech Kusa, Tommaso Furlanello, Rio Yokota, Niklas Muennighoff, Suhas Pai, Tosin Adewumi, Veronika Laippala, Xiaozhe Yao, Adalberto Junior, Alpay Ariyak, Aleksandr Drozd, Jordan Clive, Kshitij Gupta, Liangyu Chen, Qi Sun, Ken Tsui, Noah Persaud, Nour Fahmy, Tianlong Chen, Mohit Bansal, Nicolo Monti, Tai Dang, Ziyang Luo, Tien-Tung Bui, Roberto Navigli, Virendra Mehta, Matthew Blumberg, Victor May, Huu Nguyen, Sampo Pyysalo
March 30, 2024
11 citations
Key Takeaways
-
Continual training can preserve existing capabilities: Aurora-M demonstrates that adding 435B new tokens to an existing model (StarCoderPlus) doesn't cause "catastrophic forgetting"—the model maintained performance on previously learned tasks while gaining new multilingual abilities, suggesting responsible expansion of AI systems is feasible.
-
Open-source models can incorporate safety standards: Aurora-M is described as "the first open-source multilingual model fine-tuned on human-reviewed safety instructions," indicating that democratizing AI access doesn't require sacrificing alignment with safety frameworks like the Biden-Harris Executive Order on AI safety.
-
Multilingual AI development requires intentional design: The model was specifically trained on six languages (English, Finnish, Hindi, Japanese, Vietnamese, and code), suggesting that equitable AI capabilities across languages demands deliberate inclusion rather than defaulting to English-dominant systems.
-
Accessibility and responsibility can coexist: By building on existing open models (StarCoder) rather than training from scratch, Aurora-M reduces computational barriers to entry while maintaining safety considerations, demonstrating that democratizing AI development doesn't inherently conflict with responsible AI practices.
-
Evaluation across languages and safety is essential: The paper emphasizes testing "across a wide range of tasks and languages" with particular attention to "safety evaluations," suggesting responsible AI requires comprehensive, multilingual assessment rather than single-language benchmarking.