Data and Parameter Scaling Laws for Neural Machine Translation
Mitchell A. Gordon, Kevin Duh, Jared Kaplan · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · 2021
We observe that the development crossentropy loss of supervised neural machine translation models scales like a power law with the amount of training data and the number of non-embedding parameters in the model.We discuss some practical implications of these results, such as predicting BLEU achieved by large scale models and predicting the ROI of labeling data in low-resource language pairs.