Leveraging Concept-Driven Pre-Training Model for Shot-Text Conceptualization Task
Xiaoling Zhu, Yashen Wang, Yi Zhang, Xuecheng Zhang · 2023
The conceptualization of short-texts is playing an increasingly important role in text comprehension and other applications, with the aim of mapping a short-text to a set of pre-defined concepts. This work investigates and measures the adaptations of pre-training driven model in short-text conceptualization task. To this end, we discreetly design a flexible architecture as well as the corresponding procedure flow, to absorb and adopt two objectives (Concept Association Reasoner (CAR) objective and Masked-Concept Language Model (MCLM) objective respectively) into conventional pre-training architecture (e.g., BERT tested here), which have been verified on other Natural Language Processing domains. Finally, we evaluate the proposed model using several standard short-text conceptualization datasets and metrics, and experimental results showed that our model outperformed the most advanced models on real-world datasets. Especially, empirical results verify the contributions of disambiguation released by the aforementioned objectives related with concept semantics.