A Multiplication by a Neural Network (NN) with Power Activations and a Polynomial Enclosure for a NN with PReLUs
Kazuya Ozawa, Kaito Isogai, Toshihiro Tachibana, Hideo Nakano, Hideaki Okazaki · 2019
Since a series of successes of deep neural networks (DNNs) with rectified linear units (ReLUs), many approximations by NNs with ReLUs, parametric rectified linear units (PReLUs), or rectified power units (RePUs) have been focused on. However how to obtain the parameters of NNs with PReLUs, approximating polynomials, have not been fully discussed. In this paper for finding such parameters, a multiplication of NNs with n-th power (P) activation and a polynomial enclosure of NN with PReLUs are discussed. Theorem of Multiplication of n variables by a NN with n-th P, and m-th P enclosure of PReLU are provided. PReLU NN is compared with the polynomial implying the enclosure. Learnings of the approximation by the shallow, or the deep PReLU NNs are also discussed.