Augmented Mutual Information for Multi-word Extraction
Wen Zhang, Taketoshi Yoshida, Tu Bao Ho, Xijin Tang · Institutional Repositories DataBase (IRDB) · 2009
In order to extract multi-words from documents, mutual information (MI), as a statistical method, is the most popular solution under consideration. However, there are two kinds of deficiencies inherent in MI. One is the problem of unilateral cooccurrence, and the other is rare occurrence problem. To attack these two problems, augmented mutual information (AMI) is proposed in this paper to measure word dependency for multi-word extraction. We prove theoretically that AMI has the capacity to approximate MI to capture the independency of individual words, but it will amplify the significance of dependent individual words which may be possible multi-words. And ourexperimental results on Chinese multi-word extraction demonstrate that AMI method hassuperior performance to traditional MI method.