Equivalence of Empirical Risk Minimization to Regularization on the Family of $f- \text{Divergences}$
Francisco Daunas, Iñaki Esnaola, Samir M. Perlaza, H. Vincent Poor · 2024
The solution to empirical risk minimization with$f-\mathbf{divergence}$regularization$(\mathbf{ERM}-f\mathbf{DR}$) is presented under mild conditions on$f$. Under such conditions, the optimal measure is shown to be unique. Examples of the solution for particular choices of the function$f$are presented. Previously known solutions to common regularization choices are obtained by lever-aging the flexibility of the family of$f-\mathbf{divergences}$, These include the unique solutions to empirical risk minimization with relative entropy regularization (Type-I and Type-II). The analysis of the solution unveils the following properties of$f-\mathbf{divergences}$when used in the ERM-f DR problem:$i$)$f-\mathbf{divergence}$regularization forces the support of the solution to coincide with the support of the reference measure, which introduces a strong inductive bias that dominates the evidence provided by the training data; and ii) any$f-\mathbf{divergence}$regularization is equivalent to a different$f-\mathbf{divergence}$regularization with an appropriate transformation of the empirical risk function.