Unsupervised Learning of Sparse and Invariant Features Hierarchies
Y-Lan Boureau, Fu Jie Huang, Yann LeCun · 1998
Unsupervised learning methods are commonly used to produce feature extractors in image analysis systems. A challenging question is whether these methods can learn invariant hierarchies of features. This would make much easier the problem of extracting useful information from very high dimensional datasets with few labeled samples, as it is often the case in many object recognition tasks in computer vision. The feed-forward, multi-stage Hubel and Wiesel architecture [1,2,3,4,5] stacks multiple levels of alternating convolutional feature detectors, and local pooling of feature maps using some weighted average of units within a neighborhood. These models have been successfully applied to handwriting recognition [1,2], and generic object recognition [4,5]. Learning features in existing models consists in handcrafting the first layers and training the upper layers by recording templates from the training set, which leads to inefficient representations [4,5], or in training the entire architecture supervised, which requires large training sets [2,3]. In all these models, invariance is never taken into account while learning the features, but might be achieved after training by using the pooling layers [6]. We propose a fully unsupervised algorithm for learning hierarchies of sparse and locally shift-invariant features. At each level, there are multiple convolution filters followed by a max-pooling layer within a spatially local neighborhood, and a sigmoid non-linearity. Training is performed level by level, separately.