Low Carbon Footprint Training for 1D-CNNs with Temporal Max-Pooling
Anandharaju Durai Raju, Ke Wang · 2024
Training convolutional neural networks (CNNs) demands huge GPU memory consumption and training time, leading to increased carbon emissions, and impacting sustainability. In this paper, we propose HotConv, a low GPU memory and low carbon footprint learning strategy for training the class of 1D CNNs that have a temporal max-pooling layer. Such CNNs are widely used in various domains for learning large-sized inputs, including genomics and malware detection. HotConv reduces the GPU memory usage of such CNNs by harnessing the sparsity of relevant activations and gradients at the temporal max-pooling layer, which produces the same model as the full computation of activations and gradients, without trading-off model performance. Evaluations using the public benchmark BODMAS and VirusTotal datasets for malware detection with HotConv applied to the public MalConv network architecture show that the carbon footprint reduction using HotConv is superior to existing approaches. For instance, HotConv uses only 1/22 of the GPU memory used by MalConv2 - the memory-efficient variant of MalConv, while also consuming less training time than MalConv2. This is equivalent to reducing the carbon footprint up to 1/4 of that of MalConv2 without compromising performance.