Achieving External Validity in Small Domain Estimation and Causal Inference
Katherine T Li · Deep Blue (University of Michigan) · 2024
The concept of external validity is to make accurate estimates of target population quantities using a sample of the population. To conduct correct statistical inference, we must also obtain accurate standard error (SE) estimates that reflect the true sampling variation of the estimator. Achieving external validity is complicated by complex sampling designs. This thesis focuses on two specific external validity problems: small domain estimation (SDE) of substantive outcome measures---either of a certain sociodemographic subgroup or geographic area---and causal inference of treatment or exposure effects. The literature often assumes that the analyzed sample is representative of the target population and ignores data collection. Without either randomization of treatment or of selection, biases can be induced to SDE or causal effects. However, accounting for complex sampling design features is a challenge: to obtain unbiased estimates, the model must be correctly specified and incorporate design variables including survey weights. This thesis develops statistical methods to improve the external validity of SDE and causal inference by integrating multiple data sources and leveraging sampling design characteristics. In Chapter 2, we extend the multilevel regression and poststratification (MRP) framework such that we can obtain inferences for subdomains that are partially defined by variables that are available in the sample only. MRP stabilizes small domain estimates by fitting multilevel models and adjusts for selection bias by poststratifying on auxiliary variables, which are population characteristics predictive of the analytic outcome. However, its use is limited by the availability of the joint distribution of the auxiliary variables. By embedding an additional step that estimates the full poststratifier joint distribution, we can correct for the bias that is incurred by omitting the incomplete poststratifying variable from the classical MRP estimation procedure. In Chapter 3, we compare two methods for obtaining small area prevalence estimates (SAE) of a binary outcome when the sample has complex sampling design characteristics (weights, clusters, and strata). The SAEs are typically obtained via predictions from generalized linear mixed models (GLMMs) to increase estimation precision. However, obtaining the SAE variance is tricky as there is no closed-form equation for estimation, aspects that are further complicated by a complex sampling design. We compare two methods for SAE: using a weighted GLMM with jackknife replication to account for sampling design, and using an unweighted GLMM with the weighted finite population Bayesian bootstrap (WFPBB) to account for the sampling design. In Chapter 4, we obtain the population average treatment effect (PATE) from observational studies in contexts where the study sample is a subset of the target population. We focus on comparing two methods: Augmented Inverse Probability Weighting (AIPW) and Penalized Spline of Propensity Methods for Treatment Comparison (PENCOMP). These are both “doubly-robust” procedures in that consistent PATE can be obtained as long as either the treatment or outcome models are correctly specified. However, these methods lose their double-robustness property when the sample is not a simple random sample and the selection mechanism is associated with the treatment mechanism. We show that PENCOMP can be doubly-robust in this case as long as the selection weights are incorporated in the outcome model, and is considerably more efficient than equivalent AIPW methods that also account for complex sample designs.