Scaling Law for Document Neural Machine Translation
Zhang Zhuocheng, Shuhao Gu, Min Zhang, Yang Feng · 2023
The scaling laws of language models have played a significant role in advancing large language models.However, the scaling law of document-level neural machine translation remains unclear.In order to promote the development of document-level neural machine translation, we systematically examine the scaling laws in this field.In this paper, we carry out an in-depth analysis of the influence among three factors on translation quality: model scale, data scale, and maximum sequence length.Our results indicate that all three factors have a significant impact on model performance.In particular, increasing the maximum sequence length effectively reduces the context-related errors and improves the overall translation quality.Nevertheless, the sequence length cannot be increased indefinitely, as the number of parameters limits the optimal sequence length.Specifically, we propose a formula describing the empirical scaling law between the model size and the optimal sequence length.Our further analysis shows that the error accumulation problem is the primary factor that hindering further improvement in translation quality for the document-level translation by extending the sequence length.