In Defense of Knowledge Distillation for Task Incremental Learning and Its Application in 3D Object Detection
Yun Peng, Yuxuan Liu, Ming Liu · IEEE Robotics and Automation Letters · 2021
Making robots learn skills incrementally is an efficient way to design real intelligent agents. To achieve this, researchers adopt knowledge distillation to transfer old-task knowledge from old models to new ones. However, when the length of the task sequence increases, the effectiveness of knowledge distillation to prevent models from forgetting old-task knowledge degrades, which we call thelong-sequence effectiveness degradation (LED)problem. In this letter, we analyze theLEDproblem in the task-incremental-learning setting, and attribute it to the inevitable data distribution differences among tasks. To address this problem, we propose to correct the knowledge distillation for task incremental learning with a Bayesian approach. It additionally maximizes the posterior probability related to the data distributions of all seen tasks. To demonstrate its effectiveness, we further apply our proposed corrected knowledge distillation to 3D object detection. The comparison between the results of increment-at-once and increment-in-sequence experiments shows that our proposed method solves theLEDproblem. Besides, it reaches the upper-bound performance in the task-incremental-learning experiments on theKITTIdataset. The code and supplementary materials are available athttps://sites.google.com/view/c-kd/.