Bias—Variance Trade‐off
Steven W. Knox · Wiley series in probability and statistics · 2018
One can view ridge regression as an early method which exploited the bias-variance trade-off, reducing variance while increasing bias in order to produce lower-risk predictions. This chapter presents a fihure illustrating bias and variance in the k-nearest-neighbors classifier, trained on three different 150-point datasets drawn from the data source. It may be that formal extension of the bias-variance trade-off actually requires more terms than just intrinsic risk, bias, and variance: James considers this possibility in the case of symmetric loss functions. The bias and variance of an approximation method extend intuitively to learning in general with an arbitrary loss function. For arbitrary loss, the bias-variance trade-off is the idea, that finding a minimum-risk approximation method involves striking a balance between minimizing bias and minimizing variance. The risk of an approximation method decomposes in an informative way when squared-error loss is used.