Interval Type-2 Fuzzy Systems as Deep Neural Network Activation Functions
Aykut Beke, Tufan Kumbasar · 2019
In this paper, we propose a novel activation function, namely, Interval Type-2 (IT2) Fuzzy Rectifying Unit (FRU), to improve the performance of the Deep Neural Networks (DNNs).The IT2-FRU can generate linear or sophisticated activation functions by simply tuning the size of the footprint of uncertainty of the IT2 Fuzzy Sets.The novel IT2-FRU also alleviates vanishing gradient problem and has a fast convergence rate since it pushes the mean activation to zero by allowing the negative outputs.In order to test the performance of the IT2-FRU, comparative experimental studies are performed on the CIFAR-10 dataset.IT2-FRU is compared with widely used conventional activation functions.Experimental results show that IT2-FRU significantly speeds up the learning and has a superior performance compared to other handled activation functions.