Measuring Relative Accuracy of Malware Detectors in the Absence of Ground Truth
John R. H. Charlton, Pang Du, Jin-Hee Cho, Shouhuai Xu · 2018
In this paper, we measure the relative accuracy of malware detectors in the absence of ground truth regarding the quality of malware detectors (i.e., the detection accuracy) or the class of sample files, (i.e., malicious or benign). In particular, we are interested in measuring the ordinal scale of mal ware detectors in the absence of the ground truth of their actual detection quality. To this end, we propose an algorithm to estimate the relative accuracy of the malware detectors. Based on synthetic data with known ground truth, we characterize when the proposed algorithm leads to accurately estimating the relative accuracy of the malware detectors. We show the measured relative accuracy of real-world malware detectors using our proposed algorithm based on a real dataset consisting of 10.7 million files and 62 malware detectors, obtained from VirusTotal.