FlexCCT: A Methodological Framework and Software for Ratings Analysis and Wisdom of the Crowd Applications
Stephen L. France, Mahyar Sharif Vaghefi, William H. Batchelder · IEEE Transactions on Computational Social Systems · 2018
Flexible cultural consensus theory (FlexCCT) provides an integrated framework and tool set for analyzing and aggregating ratings. It utilizes a likelihood-based statistical model to create aggregate ratings weighted for rater competencies and rater biases. It has features for the analysis of multiple rating cultures and for consensus adjusted reliability. Multiple optimization algorithms are implemented for FlexCCT, along with a range of model identifiability options to restrict certain subsets of the model parameters. Bootstrapping- and jackknifing-based methods are implemented to give confidence intervals for parameters. A holdout validation method that generates a measure of “prediction log-likelihood” allows for the testing of solution reliability and stability. Empirical work demonstrates the utility of FlexCCT on unsupervised wisdom of the crowd problems, and an example shows how FlexCCT can be used to analyze educational grading data.