An Alternative to Restricted-Boltzmann Learning for Binary Latent Variables based on the Criterion of Maximal Mutual Information
David C. Edelman · 2019
The latent binary variable training problem used in the pre-training process for Deep Neural Networks is approached using the Principle (and related Criterion) of Maximum Mutual Information (MMI). This is presented as an alternative to the most widely-accepted ’Restricted Boltzmann Machine’ (RBM) approach of Hinton. The primary contribution of the present article is to present the MMI approach as the arguably more logically ’natural’ and logically simple means to the same ends. Additionally, the relative ease and effectiveness of the approach for application will be demonstrated for an example case.