Combinatorial designs for deep learning
Shoko Chisaki, Ryoh Fuji‐Hara, Nobuko Miyamoto · Journal of Combinatorial Designs · 2020
Abstract Deep learning is a machine learning methodology using a multilayer neural network. Let be mutually disjoint node sets (layers). A multilayer neural network can be regarded as a union of the complete bipartite graphs on consecutive two node sets and for . The edges of a bipartite graph function as weights which are represented as a matrix. The values of th layer are basically computed by multiplication of the weight matrix and values of th layer. Using mass training and teacher data, the weight parameters are estimated little by little. Overfitting (or overlearning) refers to a model that models the “training data” too well. It then becomes difficult for the model to generalize to new data which were not in the training set. The most popular method to avoid overfitting is called dropout. Dropout zeros out a random sample of activations (nodes) during the training process. A random sampling of nodes causes more irregular frequency of dropout edges. There is a similar sampling concept in the area of design of experiments. We propose a combinatorial design that drops out nodes from each layer. This design balances the edge frequencies. We analyze and construct such designs in this paper.