MDL, penalized likelihood, and statistical risk
Andrew R. Barron, Cong Xin Huang, Jonathan Q. Li, Xi Luo · 2008
We determine, for both countable and uncountable collections of functions, information-theoretic conditions on a penalty pen(f) such that the optimizer f of the penalized log likelihood criterion log 1/likelihood(f)+pen(f) has risk not more than the index of resolvability corresponding to the accuracy of the optimizer of the expected value of the criterion. If F is the linear span of a dictionary of functions, traditional description-length penalties are based on the number of non-zero terms (the lscr0norm of the coefficients). We specialize our general conclusions to show the lscr1norm of the coefficients times a suitable multiplier lambda is also an information-theoretically valid penalty.