Pixie: Preference in Implicit and Explicit Comparisons
Amanul Haque, Vaibhav Garg, Hui Nan Guo, Munindar P. Singh · 2022
We present Pixie, a manually annotated dataset for preference classification comprising 8,890 sentences drawn from app reviews.Unlike previous studies on preference classification, Pixie contains implicit (omitting an entity being compared), and indirect (lacking comparative linguistic cues) comparisons.We find that transformer-based pretrained models, finetuned on Pixie, achieve a weighted average F1 score of 83.34% and outperform the existing state-of-the-art for preference classification model (73.99%).