Automated assessment of correctness of recommendation systems
Angela Lozano, Andy Kellens, Kim Mens · 2012
Abstract—Using a concrete example, this position paper makes a case for evaluating the correctness of software recom-mendation systems in an automated way, prior to conducting user studies, in order to assess the validity of the results and ideal configuration of the system to be evaluated. I. CONTEXT Recommendation systems have many acclaimed advan-tages such as promoting reuse by reducing the effort required to use third party code, increasing awareness of design decisions, providing code completion hints, and pointing out incorrect or incomplete code. This paper relates our experience in evaluating Mendel [1]1, a recommendation system to support developers when extending or reusing object-oriented applications. In general, recommendation systems can be evaluated with respect to their usefulness, usability, or correctness. The usefulness of a recommendation tool is the degree to which it reduces the effort needed to perform certain development tasks, and is usually evaluated by asking developers to use the recommendation system while performing a certain task, and later discussing their opinions on the recommendations provided. Usability expresses how easy to use the recommendation tool is, and is usually evaluated via observational studies, sometimes accompanied by questionnaires. Finally, correctness evaluates how trustworthy a system is by measuring the degree to which the recommendations it proposes are correct (precision), as well as the amount of correct information it recommends (recall). II. ISSUES EVALUATING RECOMMENDATION SYSTEMS While user studies are most suited to assess the quality of recommendation systems, they are notoriously expensive to conduct. In addition to the difficulty of defining a realistic, non-biased empirical experiment, a major difficulty lies in finding a sufficiently large and representative set of developers that can serve as test subjects. These developers need to be willing to spend time learning to use the system, to answer questionnaires or interviews, and to participate in