Quantitative Member Checking
David F. Feldon, Kaylee Litson · 2024
This chapter argues that the qualitative practice of member checking is a valuable approach to gauging idiographic aspects of sensemaking underlying responses to multiple choice assessments and similar instruments. By asking respondents to synchronously assign Likert-style values to the degree of alignment between their personal understanding and the phrasing of the item they selected, it is possible to quantify incommensurability. Including quantitative incommensurability estimates in a MIMIC model with structured residuals permitted differentiation between residual variance attributable to idiographic incommensurability of the items and unclassified variance. It also provided an evaluation of the validity of individual items in terms of their narrow ability to assess the target construct consistently across individuals. The viability of this approach demonstrates the ways in which idiographic phenomena are an intrinsic part of the evaluation of items that can be incorporated into measurement models through the modeling and interpretation of residuals.