Making benchmarks uncheatable
Jin‐Yi Cai, Ajay Nerurkar, Min‐You Wu · 2002
The industry is full of anecdotes of benchmark "cheating" of various degrees ranging from innocent mistakes to flagrant misuse. Based on the tremendous progress made in recent years in theoretical computer science, it becomes possible to make a wide variety of existing benchmarks more accurate, resistant to tampering and more trustworthy. We propose two methods, the Secret Key method and the result verification method, to take computational advantage in judging performance claims. A realtime measurement method is also proposed to prevent cheating on performance reporting. The design and implementation of uncheatable benchmarks are presented in this paper.