Analyzing the Effects of Alpha Variation in Exponential Linear Unit Activation Function for Deep Neural Networks
Mahesh Kumar Singh, Shreyansh Mishra, Utkarsh Tiwari, Vikas Patel, Sahil Maurya · 2024
This research paper immerses into the subtle effects of varying values within the Exponential Linear Unit (ELU) activation function, altering from 1 to 0.10. Activation functions take an important role in deep learning models, impacting their training dynamics, convergence behavior, and stereotypical capabilities. Focusing specifically on ELU, which describes advantages over conventional activation functions like ReLU, this study aims to interpret the inference of adjusting its alpha parameter. Through a meticulous relative investigation, this paper presents a comprehensive analysis of ELUactivation function with varying alpha values. Firstly, we demonstrate the relative performance of ELU activation with varying alpha values against other conventional activation functions. Secondly, we analyze the relationship between alpha values and training loss, manifesting how habituation in alpha influences the convergence behaviorof neural networks. Thirdly, we analyze the concussion onvalidation loss to determine the effect of alpha values on the model’s ability to establish unseen data. Moreover, we analyze the correlation between alpha values and all-inclusive model performance, considering measures such as accuracy, precision, and recall by comprehensively.