Probabilistic Techniques and Rule Methods for New Word Discovery
Ziyan Jia · Jisuanji gongcheng · 2004
New word and phrase discovery is basic research in fields of NLP, IR and MT. The paper gives the method based on probabilistic techniques and rules for new word discovery via analyzing the current techniques of phrase extraction and combining the specialties of Chinese. This method includes the bi-gram probabilistic model, the statistical algorithm, the rich rules and rule-based algorithm for word filtering. Experiments show that this technique is fit for automatically and effectually extracting new words/phrases from large corpora.