Posterior calibration and exploratory analysis for natural language processing models
Khanh Duy Tung Nguyen, Brendan O’Connor · 2015
Many models in natural language processing define probabilistic distributions over linguistic structures.We argue that (1) the quality of a model's posterior distribution can and should be directly evaluated, as to whether probabilities correspond to empirical frequencies; and (2) NLP uncertainty can be projected not only to pipeline components, but also to exploratory data analysis, telling a user when to trust and not trust the NLP analysis.We present a method to analyze calibration, and apply it to compare the miscalibration of several commonly used models.We also contribute a coreference sampling algorithm that can create confidence intervals for a political event extraction task. 1