Pathologies of Orthodox Statistics
Tom Minka · 2001
By rejecting the use of a prior distribution over parameters, orthodox statistics is forced to focus on estimators, functions which guess parameter values, and to invent heuristics for choosing among estimators. Two popular heuristics are unbiasedness and maximum likelihood. Since these heuristics are not consistent with Bayes' rule, they are also not consistent with the axioms of common sense from which Bayes' rule is derived. Hence we expect there to be situations in which they violate common sense and indeed it is not hard to find such situations. This paper reviews a few simple, realistic scenarios where pathologies occur with either the unbiasedness heuristic or the maximum likelihood heuristic. 1 Introduction Many inference problems work like this: we observe some data and want to infer something about the process that generated it. If we have a probability distribution over possible processes, parameterized by `, then there is general agreement that Bayes' rule solves ...