Genetic Algorithms and Higher Order Perceptrons
Tim Andersen · 2003
Constructive induction, which is defined to be the process of constructing new and useful features from existing ones, has been extensively studied in the literature. Since the number of possible high order featuresfor any given leaming problem is exponential in the number of input attributes (where the order of a feature is defined to be the number of attributes of which it is composed), the main problem faced by constructive induction is in choosing which features to use out of this exponentially large set of potential features. For any feature set chosen the desirable characteristics are minimality and generalization performance. Two reasons for selecting small feature sets are: 1) pruning unneeded inputs allows for a savings in computational complexity during execution, 2) when choosing between two (or more) possible explanations for a given problem, the simplest is the one which will most likely produce the best generalization results. This paper uses a combination of genetic algorithms and linear programming techniques to generate feature sets. The genetic based search minimizes the size of the feature set while at the same time producing feature sets with good generalization acurracy. The features chosen are used as inputs to a high order perceptron network, which is trained with an interior point linear programming method. A critical element of the algorithm is the selection of an appropriate fitness and objective function to train the network. Several fitness/objective functions are tested and compared. The results show that the HOP is capable outperforming a multilayer backpropagation network.