Automatic extraction and filtration of multiword units1
Ying Liu, Zheng Tie · 2011
We use five statistical models including Dice coefficient (Dice), Φ2coefficient (Φ2), log likelihood ratio (LLR), symmetrical conditional probability (SCP), and normalized expectation(NE) to extract multiword unit candidates from patent corpus. We compare the results from five models and find the number of multiword unit candidates using NE is the most and the precision of Dice is the maximal, but the number of multiword unit candidates using Dice is the least and the precision of SCP is the minimum. Next the multiword unit candidates are filtrated using these filtration strategies including stop words, the threshold, higher frequency, first stop words, last stop words, and context entropy. After filtration, the number of multiword units using NE is the most and the precision of Dice is the maximal, but the number of multiword units using Dice is the least and the precision of SCP is the minimum. Each filtration strategy all help to identify the wrong or unreasonable multiword units and improve the precision of multiword units.