When and why are log-linear models self-normalizing?
Jacob Andreas, Dan Klein · 2015
Several techniques have recently been proposed for training "self-normalized" discriminative models.These attempt to find parameter settings for which unnormalized model scores approximate the true label probability.However, the theoretical properties of such techniques (and of self-normalization generally) have not been investigated.This paper examines the conditions under which we can expect self-normalization to work.We characterize a general class of distributions that admit self-normalization, and prove generalization bounds for procedures that minimize empirical normalizer variance.Motivated by these results, we describe a novel variant of an established procedure for training self-normalized models.The new procedure avoids computing normalizers for most training examples, and decreases training time by as much as factor of ten while preserving model quality.