EmbedTextNet: Dimension Reduction with Weighted Reconstruction and Correlation Losses for Efficient Text Embedding
Dae Yon Hwang, Bilal Taha, Yaroslav Nechaev · 2023
The size of embeddings generated by large language models can negatively affect system latency and model size in certain downstream practical applications (e.g.KNN search).In this work, we propose EmbedTextNet, a light add-on network that can be appended to an arbitrary language model to generate a compact embedding without requiring any changes in its architecture or training procedure.Specifically, we use a correlation penalty added to the weighted reconstruction loss that better captures the informative features in the text embeddings, which improves the efficiency of the language models.We evaluated Embed-TextNet on three different downstream tasks: text similarity, language modelling, and text retrieval.Empirical results on diverse benchmark datasets demonstrate the effectiveness and superiority of EmbedTextNet compared to state-ofart methodologies in recent works, especially in extremely low dimensional embedding sizes.The developed code for reproducibility is included in the supplementary material.1