Proof by experimentation?
Norbert Fuhr · ACM SIGIR Forum · 2020
Most IR experiments lack both internal and external validity. Top performance is mostly an illusion, given the lack of solid statistical evidence. Thus, reviewers should ignore performance numbers in their judgement, and pay more attention to proper methodology. Researchers need to focus more on performance analysis and prediction.