Modern and Novel Usages of CNNs
Ragav Venkatesan, Baoxin Li · 2017
Image datasets have constantly grown in sophistication. The MNIST-like datasets of the previous decade were structured and controlled to such a degree that they were not natural . The purpose of the feature extractor is to map the images to a space that is discriminative. Different datasets make a network learn different sets of filters. Among these datasets, it is only natural for people to expect that MNIST-rotated contains more general features than MNIST. The filters learned from different datasets would be similar if the datasets themselves were similar. While large and deep networks can be trained reasonably efficiently on GPUs and clusters of GPUs, they have too large a memory footprint to fit in mobile phones and other consumer devices with smaller form factors. Hinton et al. formed a new perspective of softmaxes in that the network learns implicitly that some classes are like others, even though such class-class similarities were never provided as supervision.