Qian Liu (Sea AI Lab) – Introduction to Modern LLM Pre-training: Sailor Use Case

This talk will present some key techniques in modern LLM pre-training, including scaling laws, data quality engineering, data mixture optimization, and efficient training strategies. We will use Sailor, a family of open language models (0.5B to 14B parameters) tailored for South-East Asian languages, as a case study. Sailor models are continually pre-trained on 200B-400B tokens across multiple languages including English, Chinese, and various SEA languages. We will discuss our empirical findings, challenges in multilingual model development, and lessons learned.

Qian Liu
Research Scientist
Sea AI Lab

Qian Liu is a Research Scientist at Sea AI Lab in Singapore. Before joining Sea AI Lab, he was a joint Ph.D. candidate at Beihang University and Microsoft Research Asia. His research interests encompass code generation and language models. He has published several papers in top conferences, with notable works including StarCoder, LoraHub and Sailor. Qian Liu has received several awards such as the KAUST AI Rising Star in 2024, and was nominated for the Baidu Scholarship in 2020. Additionally, he was one of the co-founders of the MLNLP community, a renowned NLP community in China.