Autoencoding Improves Pre-trained Word Embeddings
Masahiro Kaneko, Danushka Bollegala · 2020
Prior work investigating the geometry of pre-trained word embeddings have shown that word embeddings to be distributed in a narrow cone and by centering and projecting using principal component vectors one can increase the accuracy of a given set of pre-trained word embeddings.However, theoretically this post-processing step is equivalent to applying a linear autoencoder to minimise the squared ℓ 2 reconstruction error.This result contradicts prior work (Mu and Viswanath, 2018) that proposed to remove the top principal components from pre-trained embeddings.We experimentally verify our theoretical claims and show that retaining the top principal components is indeed useful for improving pre-trained word embeddings, without requiring access to additional linguistic resources or labeled data.