Appropriately Incorporating Statistical Significance in PMI
Om Damani, Shweta Ghonge · 2013
Two recent measures incorporate the notion of statistical significance in basic PMI formulation.In some tasks, we find that the new measures perform worse than the PMI.Our analysis shows that while the basic ideas in incorporating statistical significance in PMI are reasonable, they have been applied slightly inappropriately.By fixing this, we get new measures that improve performance over not just PMI but on other popular co-occurrence measures as well.In fact, the revised measures perform reasonably well compared with more resource intensive non co-occurrence based methods also.