A hierarchical Bayesian state trace analysis for assessing monotonicity while factoring out subject, item, and trial level dependencies
Patrick S. Sadil, Rosemary A. Cowell, David E. Huber · 2018
Traditional state trace analyses assess the latent dimensionality of a cognitive process by asking whether the means of two dependent variables conform to a monotonic function across a set of conditions. Recently proposed methods test whether a function’s deviation from monotonicity is statistically significant (e.g., Kalish et al. 2016, Davis-Stober et al. 2017). However, these tests assume trial-level independence between the two measures, but violations of this assumption can lead to incorrect conclusions. To address these limitations, we developed a hierarchical Bayesian model that factors out the separate roles of subject dependencies, item dependencies, and trial-level dependence via three separate bivariate normal distributions, capturing each type of dependency between the two measures. This is performed with separate models that do, or do not allow a non-monotonic relation between the condition effects (i.e., same vs. different rank orders). The Widely Applicable Information Criterion (WAIC) – a cross validation measure of model fit – is then used to assess the reliability of the model comparison, providing a statistical conclusion regarding the dimensionality of the latent psychological space. We validated this new state trace analysis technique using model recovery simulation studies, which assumed different ground truths regarding monotonicity and the direction/magnitude of the trial-level dependence. We also provide an application of this new technique to an implicit learning study that compared performance on an implicit retrieval task (forced choice recognition) versus an explicit retrieval task (cued recall).