New Approaches in the Learning of Complex-Valued Neural Networks
Vladyslav Kotsovsky, Anatoliy Ye. Batyuk, Maksym Yurchenko · 2020
We consider neural networks with complex weights and continuous activation functions. The complex generalization of the backpropagation learning algorithm is studied in the paper. We introduce new kinds of activation functions, namely, the complex modifications of the rational sigmoid and the ReLU activation function, the use of which has two main benefits. The first is that the application of these functions allows to avoid the splitting of transfer functions. The second is that they are fast to compute. In order to improve the performance and to increase the training speed we use the complex version of the modern optimizers instead of the classical techniques based on the application of gradient descent. The design of a complex-weighted neural networks for multiclass classification is also treated. The simulation results confirm the assumption that the combination of complex version of ReLU-like activation functions and Adam optimizer can considerably speed up the training of complex-valued neural networks.