Evaluating The Number of Trainable Parameters on Deep Maxout and LReLU Networks for Visual Recognition
Gabriel Castaneda, Paul Morris, Taghi M. Khoshgoftaar · 2020
Object recognition research has made notable steps since the appearance of convolutional neural networks, and many activation functions have been proposed to enhance the classification performance of these networks. Maxout networks have achieved great success in many computer vision tasks, but there is limited information on whether an increase in the number of trainable parameters can increase the performance in Leaky Rectified Linear Unit (LReLU) networks compared to maxout networks. Our experiments compare LReLU, rectified linear unit, scaled exponential linear unit, and hyperbolic tangent to four maxout variants. We evaluate ReLU and LReLU with 2x, 3x and 6x the number of filters in each convolutional layer. We also evaluate ReLU, LReLU and maxout networks with approximately the same number of trainable parameters. Under equal conditions, we found that on average, across all datasets, LReLU performs better than any of the evaluated activation functions.