The Quadrilogy for Single-Valued Predefined Units and Big Data
Klaus Krippendorff · 2022
Chapter 1 mentioned five kinds of reliability measures. Of these, replicability was the primary focus of the last four chapters. It is obtained from coincidences as is stability which it embraces computationally. This chapter expands α to measures of the remaining three kinds: accuracy, surrogacy, and decisiveness, of the quadrilogy of reliability measures. This quadrilogy became important in the course of a challenge to assess the reliability of very big data. This chapter describes some of the challenges of analyzing 7.5 million crowd-coded binary judgments. This chapter uses binary data to define these reliabilities: Replicability, as the ability to replicate the process of generating data elsewhere. Accuracy, as the degree to which coded data conform to a trusted standard. Surrogacy reverses the direction of trust and measures the degree to which a possible alternative can represent what trusted coders have in common. The measure of surrogacy is needed when evaluating the result of machine learning, selecting the best observer among several candidates, even the fit of a theory. Decisiveness is the degree to which what a group of observers have in common is shared by all. With these definitions, the chapter expands the quadrilogy from binary to nominal data and gives several numerical examples to illustrate the computations and their interpretations.