Development of a Software Prototype for Generating and Classifying Incomplete Many-Facet-Rasch Model Rating Designs
Mary R. McEwen · ScholarsArchive (Brigham Young University) · 2015
This doctoral research project created a software prototype to generate, classify, and visualize all possible (a priori unique) incomplete rating designs for eight raters and 24 rating objects, with 25%, 50%, and 75% rater coverage, for sub-designs that repeat over four, six, and eight rating objects. Additionally it produced a representative rater schedule (incidence matrix), concurrence matrix, and network graph visualization for each design class. Lastly, the project report included recommendations for processes researchers can use when creating an incomplete rating design. The main conclusions of this study are as follows: 1. Incomplete designs can vary greatly in their connectivity (linkage) and structure even when the number of raters, objects, and the number of raters-per-object is identical. 2. Visualization is extremely valuable for revealing details about the linkage or connectivity of an incomplete design. When the number of raters is not prohibitively large, a network graph of a proposed rating design should always be produced and evaluated before the rating data is collected. This enables the researcher to identify any disconnected subsets, as well as to more fully understand the structure and connectivity of their incomplete design. 3. More research is needed regarding the effect of an incomplete design (and its attributes) on the resulting measures from of a Many-Facets Rasch Model analysis, as compared to a complete, fully-crossed design. 4. Software is well suited to advance research in many of these issues, as well as to help the practitioner choose an optimal design for their specific purposes. More efforts should be made to develop and provide this sort of tool for those administrating or researching rater-mediated assessments.