MoRTy: Unsupervised Learning of Task-specialized Word Embeddings by Autoencoding
Nils Rethmeier, Barbara Plank · 2019
Word embeddings have undoubtedly revolutionized NLP.However, pre-trained embeddings do not always work for a specific task (or set of tasks), particularly in limited resource setups.We introduce a simple yet effective, self-supervised post-processing method that constructs task-specialized word representations by picking from a menu of reconstructing transformations to yield improved end-task performance (MORTY).The method is complementary to recent state-ofthe-art approaches to inductive transfer via fine-tuning, and forgoes costly model architectures and annotation.We evaluate MORTY on a broad range of setups, including different word embedding methods, corpus sizes and end-task semantics.Finally, we provide a surprisingly simple recipe to obtain specialized embeddings that better fit end-tasks.