Consistency of Bayes estimators of a binary regression function
Marc Coram, Steven P. Lalley, P. Lalley · 2006
Abstract. When do nonparametric Bayesian procedures “overfit? ” To shed light on this question, we consider a binary-regression problem in detail and establish frequentist consistency for a large class of Bayes procedures based on certain heirarchical priors, called uniform mixture priors. These are defined as follows: let ν be any probability distribution on the nonnegative integers. To sample a function f from the prior π ν, first sample m from ν and then sample f uniformly from the set of step functions from [0, 1] into [0, 1] that have exactly m jumps (i.e. sample all m jump locations and m + 1 function values independently and uniformly). The main result states that with only one exception, if a data-stream is generated according to any fixed, measurable binaryregression function f0 consistency obtains: i.e. for any ν with infinite support, the posterior of π ν concentrates on any L 1 neighborhood of f0. The only exception is that if f0 is identically 1, so that all class-2 label information is pure noise, inconsistency occurs if the tail of ν is too long. Qualitatively, this is the same as the finding of Diaconis and Freedman for a class of related priors. However, because the uniform mixture priors have randomly located jumps, they are more flexible and presumably more “prone ” to overfitting. Solution of a large-deviations problem is central to the consistency proof. 1.