Comparison between tagged corpora for the named entity task
Chikashi Nobata, Nigel Collier, Jun’ichi Tsujii · 2000
We present two measures for comparing corpora based on information theory statistics such as gain ratio as well as simple term-class frequency counts. We tested the predictions made by these measures about corpus difficulty in two domains --- news and molecular biology --- using the result of two well-used paradigms for NE, decision trees and HMMs and found that gain ratio was the more reliable predictor.