METASTRONG: Stata module for estimating the proportion of true effect sizes above or below a threshold in random-effects meta-analysis
Ariel Linden · RePEc: Research Papers in Economics · 2020
metastrong estimates evidence strength for scientifically meaningful effects in meta-analyses under effect heterogeneity (ie, a nonzero estimated variance of the true effect distribution) as proposed by Mathur and VanderWeele (2019; 2020). metastrong reports the estimated proportion of true effect sizes above or below a chosen threshold of a meaningful effect size q, together with confidence intervals derived via the bootstrap. These metrics could help identify if (1) there are few effects of scientifically meaningful size despite a “statistically significant” pooled point estimate, (2) there are some large effects despite an apparently null point estimate, or (3) strong effects in the direction opposite the pooled estimate also regularly occur (and thus, potential effect modifiers should be examined) (Mathur and VanderWeele 2019). By default, metastrong performs estimation using the "calibrated" method (Mathur and VanderWeele 2020) that extends work by Wang and Lee (2019). metastrong first calibrates the effect estimates and then computes the proportion of studies above (or below) q. This method makes no assumptions about the distribution of true effects and performs well in meta-analyses with as few as 10 studies. When the parametric option is specified, metastrong estimates the proportion of studies above (or below) q using the formulae devised by Mathur and VanderWeele (2019). This estimate is then bootstrapped to derive confidence intervals. As with the calibrated method, at least 10 individual studies should be available in the meta-analysis for these estimates to provide valid results. Point estimates produced by metastrong correspond to those produced by the function prop_stronger in the R package MetaUtility as of version 2.0.0. Confidence intervals will likely differ between packages given that Stata and R use different random number seeds for determining which studies to include in each bootstrap repetition.