Enhancing Out-Of-Vocabulary Word Representations with Large Language Models
Mingyu Kim, Sung-Ju Lee, Jongchan Choi · 2024
Word embeddings have demonstrated high performance in Natural Language Processing (NLP) tasks, but they face challenges in handling Out-Of-Vocabulary (OOV) words. OOV words that are not included in the pre-trained vocabulary, leading to degraded model performance. To deal with this problem, we combine word formation with contextual information. In this study, we propose a novel approach using Large Language Models (LLMs) to generate high-quality representations for OOV words. Our method uses LLMs to generate contextually appropriate replacement words for OOV words in a sentence. And make positive sample pairs and train the word representations of these samples to be closer to each other. This approach effectively combines morphological features and contextual information. It also has the advantage of maintaining context-agnostic performance for downstream tasks. This approach achieves high performance across diverse intrinsic and extrinsic tasks. This research is expected to improve performance on various NLP tasks including OOV, such as typos and neologisms.