A critique of word similarity as a method for evaluating distributional semantic models
Miroslav Batchkarov, Thomas Kober, Jeremy Reffin, Julie Weeds, David R. Weir · 2016
This paper aims to re-think the role of the word similarity task in distributional semantics research.We argue while it is a valuable tool, it should be used with care because it provides only an approximate measure of the quality of a distributional model.Word similarity evaluations assume there exists a single notion of similarity that is independent of a particular application.Further, the small size and low inter-annotator agreement of existing data sets makes it challenging to find significant differences between models.