Comparative Analysis of Activation Functions in Simple Convolutional Neural Networks
Tailin Song · 2024
Image classification has become essential in areas like computer vision, medical diagnostics, and self-driving cars, with major progress attributed to convolutional neural networks (CNNs). The effectiveness of CNNs is influenced not just by their design but also by the activation functions employed. These functions introduce non-linearity, allowing for the learning of intricate patterns. Although ReLU is favored for its simplicity and efficiency, it has drawbacks, including the “dying ReLU” issue. This research thoroughly examines how various activation functions—such as Sigmoid, Tanh, Leaky ReLU, and Exponential Linear Unit (ELU)—affect CNN performance in image classification. To ensure a fair comparison, we kept the network architecture constant while only changing the activation functions. Experiments were carried out using the MNIST dataset, a standard for image classification, to evaluate the impact of these functions on accuracy and learning behavior. The findings reveal notable performance variations, with each activation function presenting distinct benefits and limitations, shedding light on their specific roles in enhancing CNN performance.