Response to Mease and Wyner, Evidence Contrary to the Statistical View of Boosting, JMLR 9:131-156, 2008: And Yet It Overfits
Peter J. Bickel, Ya’acov Ritov · 2008
Galileo: God help us, I’m not half as sharp as those gentlemen in the philosophy department. I’m stupid, I understand absolutely nothing, so I’m compelled to fill the gaps in my knowledge... Sir, my branch of knowledge is still avid to know. The greatest problems still find us with nothing but hypotheses to go on. Yet we keep asking ourselves for proofs. Brecht (1980) This is a “sock it to them ” paper—though much less so than the previous versions one of us have seen. The authors argue in the paper that AdaBoost (without early stopping) is one of, if not the, most successful boosting algorithms, and they present this paper as a disproof of what the, rather amorphous community of statistical practitioners, represented by Friedman, Hastie and Tibshirani have: (i) Pointed out as remediable flaws of the original Freund-Schapire boosting algorithm; (ii) Given as remedies. Evidently, that community should be able to respond on its own. We in fact, agree with some of the hypotheses Mease and Wyner’s limited simulations lead them to, whether these are or are not embraced by statistical practitioners. But others we find dubious and unproven. Let us stress the