Entropy coding for training deep belief networks with imbalanced and unlabeled data
Jeffrey Berry, Ian Fasel, Luciano Fadiga, Diana B. Archangeli · The Journal of the Acoustical Society of America · 2012
Training deep belief networks (DBNs) is normally done with large data sets. In this work, the goal is to predict traces of the surface of the tongue in ultrasound images of the mouth during speech. Performance on this task can be dramatically enhanced by pre-training a DBN jointly on human-supplied traces and ultrasound images, then training a modified version of the network to predict traces from ultrasound only. However, hand-tracing the entire dataset of ultrasound images is extremely labor intensive. Moreover, the dataset is highly imbalanced since many images are extremely similar. This work presents a bootstrapping method which takes advantage of this imbalance, iteratively selecting a small subset of images to be hand-traced, then (re)training the DBN, making use of an entropy-based diversity measure for the initial selection. With this approach, a three-fold reduction in human time required to trace an entire dataset with human-level accuracy was achieved.