CLongEval: A Chinese Benchmark for Evaluating Long-Context Large Language Models
Zexuan Qiu, Jingjing Li, Shijue Huang, Xiaoqi Jiao, Wanjun Zhong, Irwin King · 2024
Developing Large Language Models (LLMs) with robust long-context capabilities has been the recent research focus, resulting in the emergence of long-context LLMs proficient in Chinese.However, the evaluation of these models remains underdeveloped due to a lack of benchmarks.To address this gap, we present CLongEval, a comprehensive Chinese benchmark for evaluating long-context LLMs.CLongEval is characterized by three key features: (1) Sufficient data volume, comprising 7 distinct tasks and 7,267 examples; (2) Broad applicability, accommodating to models with context windows size from 1K to 100K; (3) High quality, with over 2,000 manually annotated question-answer pairs in addition to the automatically constructed labels.With CLongEval, we undertake a comprehensive assessment of 6 open-source long-context LLMs and 2 leading commercial counterparts that feature both long-context abilities and proficiency in Chinese.We also provide in-depth analysis based on the empirical results, trying to shed light on the critical capabilities that present challenges in long-context settings.1