Automated Assessment of Secure Search Systems
Mayank Varia, Benjamin Wills Price, Nicholas J. Hwang, Ariel Hamlin, Jonathan C. Herzog, Jill Poland, Michael Reschly, Sophia Yakoubov, Robert K. Cunningham · ACM SIGOPS Operating Systems Review · 2015
This work presents the results of a three-year project that assessed nine different privacy-preserving data search systems. We detail the design of a software assessment framework that focuses on low system footprint, repeatability, and reusability. A unique achievement of this project was the automation and integration of the entire test process, from the production and execution of tests to the generation of human-readable evaluation reports. We synthesize our experiences into a set of simple mantras that we recommend following in the design of any assessment framework.