Do We Really Need a New Theory to Understand the Double-Descent?
Luca Oneto, Sandro Ridella, Davide Anguita · 2022
This century saw an unprecedented increase of public and private investments in Artificial Intelligence (AI) and especially in Machine Learning (ML).This led to breakthroughs in their practical ability to solve complex real world problems impacting research and society at large.Instead, our ability to understand the fundamental mechanism behind these breakthroughs has slowed down because of their increased complexity.This questioned researchers about the necessity for a new theoretical framework able to help researchers catch up on this lag.One of the still not well understood mechanisms is the so called over-parametrization, namely the ability of certain models to increasing their generalization performance (reduce test error) when the number of parameters is above the interpolating threshold (zero training error), and the associated doubledescent curve.In this paper we will show that this phenomena can be better understood using both known theories, i.e., the algorithmic stability theory, and empirical evidence.