SCORE CORRECTION FOR BIASED GRADERS IN A NETWORK
Calvin Li · 2016
A method of convex optimization for taking into account biased graders in a network is disclosed. The method involves the steps of creating a convex function, determining the grade of the object and then finding bounds for the convex function. This method of convex optimization could be applied in a number of applications where multiple graders grade multiple consistent objects. BACKGROUND While determining a score for an object in a system where the quality of the graders is variable, the resulting score obtained by taking the average of the score provided by the graders would be unreliable and inconsistent. There can be the likes of good graders, bad graders and lenient graders, or harsh graders. For example, object A could receive a lower score and can be ranked lower even when the objective score of object A is higher than that of object B. This can happen when object A receives strict graders while object B receives lenient graders. The results can also be random if both the objects receive graders of poor quality. Therefore, an ideal solution would be to have all objects in the system graded by the same person, so that there is consistency in the grades. However, this becomes problematic in terms of resources as there are either way too many graders or way too many objects to score, thus making it economically or practically intractable. Recently, for systems with large pool of objects and few graders per object or vice versa, the common approach is to average the grades on the objects without considering any network effects. Similarly, for systems with a small pool of objects, the size is tractable enough where there are enough graders to grade objects or the graders know each other to form consistent assessment. Thus there is a need for a better method to solve the grader problem. 2 Li: SCORE CORRECTION FOR BIASED GRADERS IN A NETWORK Published by Technical Disclosure Commons, 2016 DESCRIPTION This disclosure presents a method to use convex optimization approach for biased graders in a network to optimize a convex problem. The method of convex optimization involves the steps of creating a convex function, determining the grade of the object and then finding bounds for the convex function. (i) Creation of a convex function The convex optimization technique has many variants and works by creating a convex function to map the initial score of the grade to the final score. Convex problems have the niche property of being able to extrapolate local maxima to global maxima by allowing optimal values to be determined in a seemingly complicated-looking optimization problem. The convex function is thus written as: (s_j_i) = where, f_j is the convex function for grader j, s_j_i is the score grader j would give object i, t_i is the grade of object i. This model calculates the grade by taking the average grade of various graders. This assumption can be explored to create a better model when challenged. A simple convex solution to this would be: (x) = x + b_j where, b_j is the bias that a grader has. Finding the value of b_j would determine how different the mean of the scores provided by the grader is from the scores. A more complex convex function would be: (x) = a_j * x + b_j This convex function would determine the following three aspects: 1. Bias of the mean. 3 Defensive Publications Series, Art. 182 [2016] http://www.tdcommons.org/dpubs_series/182 2. Spread of the grades relative to scores, i.e., the size of a grader's standard deviations on his/her score. 3. Consistency of the grader (i.e how correlated is the grader with the scores). For example, a grader with a negative a_j is a terrible grader as his/her grades are inversely correlated with the true score. (ii) Determining grade of an object (t_i) Since the assumption of this technique is that the score is the average of all the graders' score on one object, values a and b that would minimize the differences in the interpretation of the values need to be determined. Thus, the function of convex optimization would be to determine the values of a and b using minimize: p-norm of (f_j(s_i_j) u_i), where, p-norm is a mathematical concept that is convex and p is a number from 1 to infinity and u_i is the average of all f (s_i) (average of all graders who have graded object i). Choosing 1 would minimize the sum of the differences (min sum), and choosing infinity would minimize the highest value (min max). Numbers falling in between these two ranges would give a mix of the two said objectives and hence 2 norms having smooth properties are commonly used. With the variability of the graders and the flexibility of the model used to represent f, it is highly improbable that nontrivial a's and b's can be chosen that would minimize the noise. Therefore, it is necessary to minimize the discrepancy between the values. (iii) Finding bounds of the convex function Finding bounds solves four problems viz., prevents the problem space becoming convex, accurately reflects the reality of the model space, prevents trivial solutions from arising and finally prevents the model from overfitting. 4 Li: SCORE CORRECTION FOR BIASED GRADERS IN A NETWORK Published by Technical Disclosure Commons, 2016 Using (x) = a_j * x + b_j as an example, bounds (x) would be between the spectrum of values. Here, there is a disadvantage of really good objects and poor objects becoming indistinguishable near the extremes. Thus a different convex function will be able to make the bounds exponential. 1. bound a to prevent a trivial solution. a != 0 is not a valid bound, but Sum of a = some value is a valid bound. a > 1 is also a valid bound. 2. bounding b is pointless and can be obtrusive if f_j(x) is already bounded. This optimization technique tries to learn the graders' skill relative to the rest of the population as they engage with other graders through common graded objects. The advantage of the method is that it does not require external information other than what is fundamentally available to the grading problem (i.e. who graded it and what object was graded by them). However the method requires the graders to be connected. Paired grading for the duration of the grading experience would render this model useless as it would require more data to figure out the quality of the grader than what the model depends on. To use a networked grading method, a lot of external information is needed where the graders go through a rigorous test and review. Convex optimization can be applied in a number of applications where multiple graders grade multiple consistent objects, for example, restaurant review, interview feedback and score and distributed paper grading between multiple graders and reviewers. 5 Defensive Publications Series, Art. 182 [2016] http://www.tdcommons.org/dpubs_series/182