Research on Chinese New Word Discovery Algorithm Based on Mutual Information
Gaohui Shang · 2019
For the existing Chinese new word discovery method, some fixed expressions that are not phrases or non-new words will be recognized as new words. Aiming at this problem, a new word discovery algorithm based on the principle of similarity judgment is proposed. The principle of similarity judgment is that if the context of a candidate substring is similar to the context of another expression, the probability of it becoming a new word will increase. The new algorithm is based on the unsupervised new word discovery algorithm filtered by statistical information such as mutual information, information entropy, and word frequency. The similarity and mutual information are combined into similarity enhancement mutual information, which is used as an indicator to measure internal integration. Finally, experiments show that the algorithm of similarity enhancement mutual information can effectively identify new words in the industry with high precision.