A Turing Test for B-Keeper: Evaluating an Interactive Real-Time Beat-Tracker
Andrew Robertson, Mark D. Plumbley, Nick Bryan–Kinns · Surrey Research Insight Open Access (The University of Surrey) · 2008
Off-line beat trackers are often compared to human tappers who provide a ground truth against which they can be judged. In order to evaluate a real-time beat tracker, we have taken the paradigm of the ‘Turing Test’ in which an interrogator is asked to distinguish between human and machine. A drummer plays in succession with an interactive accompaniment that has one of three possible tempocontrollers (the beat tracker, a human tapper and a steadytempo metronome). The test is double-blind since the researchers do not know which controller is currently functioning. All participants are asked to rate the accompaniment and to judge which controller they believe was responsible. This method for evaluation enables the controllers to be contrasted in a more quantifiable way than the subjective testimony we have used in the past to evaluate the system. The results of the experiment suggest that the beat tracker and a human tapper are both distinguishable from a steadytempo accompaniment and they are preferable according to the ratings given by the participants. Also, the beat tracker and a human tapper are not sufficiently distinguishable by any of the participants in the experiment, which suggests that the system is comparable in performance to a human tapper.