An Improved Algorithm for Bert
Weiguo Wang, Zhaoquan Gu, Keke Tang · 2020
Natural language processing (NLP) has been widely applied in machine translation, speech recognition and other fields. Many elegant results have been proposed such as text classification, sentiment analysis. In recent years, the Bert model has made a great breakthrough in many NLP tasks. Afterwards, using transformer as a feature extractor has been achieving good performance in many scenarios. However, there are too many model parameters in the Bert model. In order to solve this problem, we propose an improved algorithm for the Bert model, which shares the first 11 layers' parameters. The proposed method could largely reduce the number of parameters in the model. We also conduct experiments to evaluate the algorithm's performance; the results also validate that the improved algorithm for the Bert model could achieve good results.