The Four Ways to Consider Measurement Noise in Bayesian Model Selection—And Which One to Choose
Sebastian Reuschen, Wolfgang Nowak, Anneli Guthke · Water Resources Research · 2021
Abstract Bayesian model selection (BMS) is a statistically rigorous approach to assess the plausibility of competing models. It naturally accounts for uncertainties in models and data. In this study, we discuss the role of measurement noise in BMS deeper than in past literature. We distinguish between four cases, accounting for noise in models and/or data: (1) no‐no, (2) no‐yes, (3) yes‐no, and (4) yes‐yes. These cases differ mathematically and philosophically. Only two out of these four cases are logically consistent, and they represent two potentially conflicting research questions: “Which model is best in modeling the pure physics?” (Case 1) and “which model is best in predicting the data‐generating process (i.e., physics plus noise)?” (Case 4). If we are interested in the “pure physics question,” we face two practical challenges: First, we would need noise‐free data, which is impossible to obtain; and second, the numerical approximation of Bayesian model evidence can be hard when neglecting noise. We discuss how to address both challenges and reveal that a fallback to the easier “data‐generation question” as a proxy for the “physics question” is not appropriate. We demonstrate on synthetic scenarios and a real‐world hydrogeological case study that the choice of the case has a significant impact on the outcome of posterior model weights, and hence on results of the model ranking, model selection, model averaging, model confusion analysis, and uncertainty quantification. Reality might force us to use a different case than philosophy would suggest, and we provide guidance on how to interpret model probabilities under such conditions.