EDAW: Enhanced Knowledge Distillation and Adaptive Pseudo Label Weights for Continual Named Entity Recognition
Yubin Sheng, Zuping Zhang, Panrui Tang, Bo Huang, Xiao Lan Yao · 2024
Continual Learning for Named Entity Recognition (CL-NER) is designed to train models capable of adapting to evolving data by continuously introducing new entity types. This approach is crucial in dynamic environments where data evolves, such as social media, healthcare, and legal documents, necessitating the model to retain the memory of previously learned entity types while learning to identify new ones. However, due to the neural network's tendency to acquire new knowledge and forget old knowledge in continual learning and the unique non-entity type annotations in NER tasks, CL-NER faces severe catastrophic forgetting and semantic drift issues. In this paper, we propose Enhanced Knowledge Distillation and Entropy-based Adaptive Pseudo Label Weights (EDAW) to address the catastrophic forgetting and semantic drift issues in CL-NER. Specifically, we develop an enhanced knowledge distillation method that combines Kullback-Leibler divergence and feature cosine discrepancy. This method effectively minimizes the variance in output probability distributions and aligns the internal feature spaces between new and old models, thus reducing catastrophic forgetting. Additionally, we propose an entropy-based adaptive pseudo label weight method that allows the model to assign different weights to pseudo labels with varying certainties during training, effectively alleviating semantic drift and error accumulation caused by erroneous relabeling of pseudo labels. Notably, this study pioneers the in-clusion of a Chinese dataset in CL-NER, enhancing the model's robustness and demonstrating its efficacy in a multilingual context. Experiments on fourteen CL-NER settings across four public NER datasets show that EDAW improves average Micro-F1 and Macro-F1 scores by 3.44% and 3.72%, respectively, over existing state-of-the-art(SOTA) methods. We make our code available at: https://github.com/livosr/EDAW/tree/master