Developing Prefix-Tuning Models for Hierarchical Text Classification
Lei Chen, Hou Wei Chou, Xiaodan Zhu · 2022
Hierarchical text classification (HTC) is a key problem and task in many industrial applications, which aims to predict labels organized in a hierarchy for given input text.For example, HTC can group the descriptions of online products into a taxonomy or organizing customer reviews into a hierarchy of categories.In real-life applications, while Pre-trained Language Models (PLMs) have dominated many NLP tasks, they face significant challenges too-the conventional fine-tuning process needs to modify and save models with a huge number of parameters.This is becoming more critical for HTC in both global and local modelling-the latter needs to learn multiple classifiers at different levels/nodes in a hierarchy.The concern will be even more serious since PLM sizes are continuing to increase in order to attain more competitive performances.Most recently, prefix tuning has become a very attractive technology by only tuning and saving a tiny set of parameters.Exploring prefix turning for HTC is hence highly desirable and has timely impact.In this paper, we investigate prefix tuning on HTC in two typical setups: local and global HTC.Our experiment shows that the prefix-tuning model only needs less than 1% of parameters and can achieve performance comparable to regular full fine-tuning.We demonstrate that using contrastive learning in learning prefix vectors can further improve HTC performance.