Feature distillation with mixup for unsupervised continual learning
Yiyan Gao · 2023
We learn feature representations on unlabeled task sequences. In order to make the disparity between unsupervised representational learning and continuous learning narrow, interpolate examples of current and previous tasks to build new examples. On this basis, we suggest a innovative knowledge extraction method which takes into account feature location and distance function of the extraction. We use the position before ReLU as the distillation point and design a new margin ReLU function. This allows for the centralization of useful information in the middle of the network and further performance improvement, minimizing the forgetting of past activities while increasing learning of new ones.