Reexamining the principle of mean-variance preservation for neural network initialization
Kyle Luther, H. Sebastian Seung · Physical Review Research · 2020
This paper examines neural network parameter initialization schemes by distinguishing between two sorts of randomness: randomness in network parameters and randomness in network inputs. The authors show that in higher layers of a deep network fluctuations arising from input randomness can decay to zero while the scale of fluctuations arising from parameter randomness remains constant.