Ancient Tibetan Word Segmentation based on Deep Learning
Bo An, Congjun Long · 2021
Tibetan ancient literature is an important literature material for the study of ancient Tibetan culture, history and language, which has important academic value for the study of the development of Sino Tibetan language family. However, the lack of research on ancient Tibetan word segmentation seriously restricts the research of ancient Tibetan literature. In view of this situation, this paper utilizes ancient Tibetan interlaced contrast tagging data to extract the ancient Tibetan word segmentation data set. Based on this dataset, this paper conduct the research on dictionary based word segmentation, statistics based word segmentation and deep learning based ancient Tibetan word segmentation. Experimental results show that BiLSTM + CRF word segmentation algorithm can achieve the best performance by a single model, and the effect of ancient Tibetan word segmentation can be further improved through model ensemble. And the results show that the unknown words, insufficient training data and word ambiguity also restrict the performance of ancient Tibetan word segmentation.