A consistent estimator of the expected gradient outerproduct
Shubhendu Trivedi, Jialei Wang, Samory Kpotufe, Gregory Shakhnarovich · 2014
In high-dimensional classification or regression problems, the expected gradient outerproduct (EGOP) of the unknown regression function f, namely Ex (∇ f(X) · ∇f(X)⊤), is known to recover those directions v ∈ ℝd most relevant to predicting the output Y. However, just as in gradient estimation, optimal estimators of the EGOP can be expensive in practice. We show that a simple rough estimator, much cheaper in practice, suffices to obtain significant improvements on real-world nonparametric classification and regression tasks. Furthermore, we prove that, despite its simplicity, this rough estimator remains statistically consistent under mild conditions.