Continual Learning of Deep Neural Networks in The Age of Big Data
Alexander Gepperth, Timothée Lesort · 2024
Many applications of deep learning are set in an environment with perpetual change or at least with an ever-growing amount of data.In practice, deep neural network (DNNs) and large language models (LLMs) are continually trained and evaluated.They need to incorporate new data or new annotations, where one typical issue is the extensive availability of unannotated or low-quality data, coupled with a bottleneck concerning annotations and/or curated samples.In such setups, the scaling behavior of continual learning (CL) algorithms w.r.t.training time becomes critical, which is in contrast to the standard CL setting operating on small databases like MNIST, CIFAR or ImageNet.Annotations or curated samples become available progressively, e.g., because they are created by humans, or due to an ongoing exploration of the environment, and need to be progressively incorporated into models.This article explores how advancement in continual learning can improve the scalability and performance of DNNs and LLMs in such setups.One interesting aspect is to leverage dedicated (small-scale) CL techniques to achieve advantageous trade-offs between computational cost and accuracy, or how such CL methods can maintain advantageous scaling behavior w.r.t.continuous re-training on all data.99