Enhancing Cross-Lingual Word Embeddings: Aligned Subword Vectors for Out-of-Vocabulary Terms in fastText
Claudio Savelli, Flavio Giobergia · 2024
The word embedding models have significantly impacted Natural Language Processing, particularly in multilingual settings. The well-known fastText approach proposed a straightforward and revolutionary way to generate embeddings for out-of-vocabulary (OOV) words by using subword information, allowing the generation of embeddings for words not present in the initial training set. In addition, other methods tried to align word embeddings across different languages, facilitating a multilingual setting that enhances cross-lingual natural language processing tasks. However, the fastText-aligned embeddings do not allow alignment of the subword information. This work addresses this challenge by enhancing the fastText model, introducing aligned subword vectors, thereby improving fastText’s capacity to handle OOV terms in a cross-lingual context. We propose a method to reconstruct the alignment matrix from known word embeddings and apply this matrix to subword vectors, enabling the generation of cross-lingually aligned embeddings for OOV terms. We then demonstrate the accuracy of our alignment matrix reconstruction and the effectiveness of our approach in producing meaningful embeddings for previously unseen words across multiple languages, as well as show the possible limitations of the aligned subword embeddings. The code and the obtained word embeddings can be found at link1.