Context-Relevant Denoising for Unsupervised Domain-Adapted Sentence Embeddings
Michael Lowe, Joseph D. Prusa, Joffrey L. Leevy, Taghi M. Khoshgoftaar · 2024
In closed-system domains, such as healthcare databases, record scarcity and data quality often act as barriers to applying state-of-the-art language processing techniques. Addressing these challenges requires the adjustment of both domain and task to effectively deliver meaningful value. A common approach for adapting domains with limited and poorly annotated data is data augmentation. Transformers and Sequential Denoising Auto-Encoders (TSDAEs) offer an inductive, unsupervised pretraining method that efficiently leverages unlabeled data by learning from many-to-one corrupted training samples. This approach reduces the need for extensive manual data annotation typically associated with domain adaptation. We advance this method by using transduction-based noise generation, which simulates the kind of noise commonly encountered in text generation within targeted domains. Our study investigates the effects of corruption and contextual noise introduced by this augmentation, thus enhancing the practical ability of domain-adapted models in specialized fields.