Should I Stay or Should I Grow? A Dynamic Self-Governed Growth for Determining Hidden Layer Size in a Multilayer Perceptron
Matt Ross, Nareg Berberian, Sylvain Chartier · 2020
A novel dynamic self-governed growth algorithm inspired from population dynamics is introduced in a Multi-Layer Perceptron (MLP). This allows the inclusion of a carrying capacity, which is the maximum population of hidden units that can be sustained in a single hidden layer. The inclusion of this constraint in combination with population dynamics provides a built-in mechanism for a dynamic growth rate. The proposed approach is used in parallel with direct performance feedback from the network to modulate the growth rate of the hidden layer. This algorithm incrementally adds units to the hidden layer up to a point where the complexity of the task no longer requires further addition. The MLP is extended with the growing algorithm and its adaptability is tested by subjecting the network to increasing levels of task complexity for the n-bit problem. Using fixed rules that dictate both the size of a fixed layer MLP (fMLP) and the upper bound carrying capacity of the growing MLP (gMLP), the resulting topologies are directly compared for the n-bit problem. In short, the results suggest that even if an upper boundary of the carrying capacity is set by a fixed rule, the growing algorithm is capable of converging to less than the predicted number of units required for solving the given task. With the majority of trials growing to the same number of hidden units regardless of the rule used. This effect is consistent across the specified rules and levels of task complexity for the n-bit problem.