Measuring Sentiment Annotation Complexity of Text
Aditya Joshi, Abhijit K. Mishra, Nivvedan Senthamilselvan, Pushpak Bhattacharyya · 2014
The effort required for a human annotator to detect sentiment is not uniform for all texts, irrespective of his/her expertise.We aim to predict a score that quantifies this effort, using linguistic properties of the text.Our proposed metric is called Sentiment Annotation Complexity (SAC).As for training data, since any direct judgment of complexity by a human annotator is fraught with subjectivity, we rely on cognitive evidence from eye-tracking.The sentences in our dataset are labeled with SAC scores derived from eye-fixation duration.Using linguistic features and annotated SACs, we train a regressor that predicts the SAC with a best mean error rate of 22.02% for five-fold cross-validation.We also study the correlation between a human annotator's perception of complexity and a machine's confidence in polarity determination.The merit of our work lies in (a) deciding the sentiment annotation cost in, for example, a crowdsourcing setting, (b) choosing the right classifier for sentiment prediction.