Should Security Researchers Experiment More and Draw More Inferences?
Kevin S. Killourhy, Roy A. Maxion · 2011
“Whenever possible, experiments should be comparative. For example, if you are testing a modification, the modified and unmodified procedures should be run side by side in the same experiment. ” Box,Hunter, and Hunter [1] “It is possible, and indeed it is all too frequent, for an experiment to be so conducted that no valid estimate of error is available. In such a case the experiment cannot be said,strictly,tobe capable of proving anything. ” R.A.Fisher [2] Two methodological practices are well established in other scientific disciplines yet remain rare in computersecurity research: comparativeexperimentsand statistical inferences. Comparative experiments offer the only way to control factors that might vary from one study to the next. Statistical inferences enable a researcher to drawgeneralconclusionsfromempiricalresults. Despite their widespread use in other sciences, these practices are haphazardly used in security research. Choosing keystroke dynamics as an example to study, we survey the literature. Of 80 papers wherein these practices would be appropriate, only 43 (53.75%) performed comparative experiments, and only 6 (7.5%) drewstatistical inferences. In disciplines such as medicine, comparative experiments and statistical inferencessave lives and cut costs. Rigorous methodological standards are required. We see no reason why security research, another discipline wherethestakesarecriticallyhigh,cannotorshouldnot adoptthese practicesaswell. Failureto takeamorescientific approach to security research stalls progress and leavesusvulnerable. 1