Top in the Lab, Flop in the Field?
Peter Widhalm, Maximilian Leodolter, Norbert Brändle · 2018
We present a solution to the Sussex-Huawei Locomotion-Transportation (SHL) recognition challenge (team "S304"). Our experiments reveal two potential pitfalls in the evaluation of activity recognition algorithms: 1) unnoticed overfitting due to autocorrelation (i.e. dependencies between temporally close samples), and 2) the accuracy/generality trade-off due to idealized conditions and lack of variation in the data. We show that evaluation with a random training/test split suggests highly accurate recognition of eight different travel activities with an average F1 score of 96% for single-participant/fixed-position data, whereas with proper backtesting the F1 score drops to 84%, for data of different participants in the SHL Dataset to 61%, and for different carrying positions to 54%. Our experiments demonstrate that results achieved 'in-the-lab' can easily become subject to an upward bias and cannot always serve as reliable indicators for the future performance 'in-the-field', where generality and robustness are essential.