On Fisher Information Matrix for Simple Neural Networks with Softplus Activation
Masazumi Iida, Yoshinari Takeishi, Jun’ichi Takeuchi · 2022 IEEE International Symposium on Information Theory (ISIT) · 2022
Fisher information of simple neural networks with the softplus activation function is argued. We show that, under certain conditions, FIMs of simple models have the similar interesting spectral structure as the one shown by Takeishi et al (2021) for networks with ReLU. This work helps us to understand why the FIM has such the structure.