LongAlign: A Recipe for Long Context Alignment of Large Language Models
Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Jing Hou, Jie Tang, Yuxiao Dong, Juanzi Li · 2024
Extending large language models to effectively handle long contexts requires instruction finetuning on input sequences of similar length.To address this, we present LongAlign-a recipe of the instruction data, training, and evaluation for long context alignment.First, we construct a long instruction-following dataset using Self-Instruct.To ensure the data diversity, it covers a broad range of tasks from various long context sources.Second, we investigate different strategies to speed up supervised fine-tuning on datasets with uneven length distribution, namely packing and sorted batching.Additionally, we develop a loss weighting method to balance the contribution to the loss across different sequences during packing training.Third, we introduce the LongBench-Chat benchmark for evaluating instruction-following capabilities on queries of 10k-100k in length.Experiments show that LongAlign outperforms existing recipes for LLMs in long context tasks by up to 30%, while also maintaining their proficiency in handling short, generic tasks.