A minimum description length framework for unsupervised learning
Richard S. Zemel · 1994
A fundamental problem in learning and reasoning about a set of information is finding the right representation. The primary goal of an unsupervised learning procedure is to optimize the quality of a system's internal representation. In this thesis, we present a general framework for describing unsupervised learning procedures based on the Minimum Description Length (MDL) principle. The MDL principle states that the best model is one that minimizes the summed description length of the model and the data with respect to the model. Applying this approach to the unsupervised learning problem makes explicit a key trade off between the accuracy of a representation (i.e., how concise a description of the input may be generated from it) and its succinctness (i.e., how compactly the representation itself can be described). Viewing existing unsupervised learning procedures in terms of the framework exposes their implicit assumptions about the type of structure assumed to underlie the data. While...