Learning to Embed Words in Context for Syntactic Tasks
Lifu Tu, Kevin Gimpel, Karen Livescu · 2017
We present models for embedding words in the context of surrounding words.Such models, which we refer to as token embeddings, represent the characteristics of a word that are specific to a given context, such as word sense, syntactic category, and semantic role.We explore simple, efficient token embedding models based on standard neural network architectures.We learn token embeddings on a large amount of unannotated text and evaluate them as features for part-of-speech taggers and dependency parsers trained on much smaller amounts of annotated data.We find that predictors endowed with token embeddings consistently outperform baseline predictors across a range of context window and training set sizes.