HSSAS: Optimizing NLP Models by Combining Self-Supervised Learning and Neural Architecture Search
Shuocheng Qi, Dong Wang, Yuling Fan, Likai Dong · 2024
With the increasingly widespread applications of Natural Language Processing (NLP), many algorithms based on deep learning have been proposed and introduced, which have achieved outstanding achievements. However, these algorithms now face challenges due to their strong dependence on labeled data and high computational costs. These issues negatively impact the generality, efficiency and scalability of the algorithms in practical applications. This study proposes a Hybrid Self-Supervised Architecture Search (HSSAS) method combining Self-Supervised Learning (SSL) and Neural Architecture Search (NAS), which can effectively alleviate the challenges of dependency on labeled data and high computational costs. Before conducting Neural Architecture Search (NAS) to explore model architectures, Self-Supervised Learning (SSL) defines specific tasks to compel the model to understand contextual relationships, enabling it to extract high-quality feature representations from unlabeled data. These features encapsulate core information of the data with high generalization ability. Traditional NAS typically requires evaluating hundreds of architectures. However, by leveraging the feature representations obtained from SSL, NAS can efficiently evaluate architectures on a condensed feature set, filtering out unsuitable options. This substantially reduces the search space, decreases computational complexity, and saves both time and resources. Experiments are carried on the BBC news dataset and the results demonstrate that the HSSAS method outperforms traditional NAS methods and NAS methods without SSL on multiple performance metrics, while also maintaining lower computational overhead.