Metrics-Based Test and Evaluation of Group Detection Software for Counter-Terrorism
T. Garwin, M.P. Crozat, B.L. Merrell, B. Bebee · 2006
Multiple intelligence databases contain volumes of link information connecting terrorism-related individuals. Software to identify active groups within this mass of data has been an important focus of research and development. This paper describes experience with an integrated set of test methods developed to evaluate the technical accuracy and real-world value of group detection algorithms as applied to link datasets of varying size and characteristics. The testing includes an automated software harness to score the algorithms against representative datasets that are generated by a controlled random process from known ground truth. It also applies novel statistical benchmarks and techniques to evaluate the value of the algorithms as applied by analysts to real intelligence data where ground-truth is absent. Together, these tests have increased understanding of the determinants of algorithmic accuracy, aided in selection among competing software packages, and resulted in significantly improved accuracy and value-in-use as a result of feedback to developers.