PiC: A Phrase-in-Context Dataset for Phrase Understanding and Semantic Search
Thang M. Pham, Seunghyun Yoon, Trung Bui, Anh‐Tu Nguyen · 2023
While contextualized word embeddings have been a de-facto standard, learning contextualized phrase embeddings is less explored and being hindered by the lack of a human-annotated benchmark that tests machine understanding of phrase semantics given a context sentence or paragraph (instead of phrases alone).To fill this gap, we propose PiC-a dataset of ∼28K of noun phrases accompanied by their contextual Wikipedia pages and a suite of three tasks for training and evaluating phrase embeddings.Training on PiC improves ranking-models' accuracy and remarkably pushes span-selection (SS) models (i.e., predicting the start and end index of the target phrase) near human-accuracy, which is 95% Exact Match (EM) on semantic search given a query phrase and a passage.Interestingly, we find evidence that such impressive performance is because the SS models learn to better capture the common meaning of a phrase regardless of its actual context.SotA models perform poorly in distinguishing two senses of the same phrase in two contexts (∼60% EM) and in estimating the similarity between two different phrases in the same context (∼70% EM).PR-pass PR-page PSD SQuAD 1.1 HotpotQA All instances 28,147 28,098 4,858 98,169 105,257 Unique queries/questions 27,055 27,016 4,812 97,888 105,249 Unique answers 13,458 13,423 2,314 72,469 57,259