Double distillation for class incremental learning
Darian M. Onchiş, Ioan-Valentin Samuila · 2021
Incremental learning models belong to an area of machine learning algorithms dealing with streams of data arriving in sequential order over time. As such, these algorithms are vulnerable to catastrophic forgetting, which is the tendency of the incremental models to forget past information when new data is added. This phenomenon could be attenuated with the use of a knowledge distillation component and with the support of a sample memory from past classes. In here, we propose a double distillation incremental learning recipe for the class incremental learning scenario, starting from a proof of the classification limits of the relaxed SoftMax function and coupling it with a modified version of the iCaRL algorithm in which we have remodeled the last classification layer by varying the temperature parameter. We performed extensive experiments to outline the advantages of using our method compared to the standard algorithm.