Measuring Programming Skill - Construction and Validation of an Instrument for Evaluating Java Developers
Gunnar Rye Bergersen · 2015
Skilled developers are important to the software industry. In empirical studies in software engineering, knowing the skill level of the participants is also important for correct interpretation of results. The current practice in industry and research for assessing programming skills is mostly to use proxy variables of skill, such as education, experience, and multiple-choice knowledge tests. There is as yet no valid and efficient way to measure programming skill. Consequently, this thesis aimed to construct a valid instrument for measuring programming skill, where skill is inferred from performance on programming tasks. The Rasch measurement model was used to construct the instrument. Sixty-five professional developers from eight countries participated in validating the instrument, solving 19 Java programming tasks over two days. The validity of the instrument was theoretically investigated through commercial and research-based tests. Programming skill, as measured by the instrument, was also investigated in terms of experience and other background variables. The instrument was found to have desirable psychometric properties, and the overall results appear well aligned with theoretical expectations. This work has shown that acceptable measures of programming skill may be obtained with less than one day of testing. Further work should be directed at reducing the time needed to measure programming skill without affecting the validity of the instrument. The results of the research have already been transferred to the industry through a commercial prototype.