TorchMetrics - Measuring Reproducibility in PyTorch
Nicki Skafte Detlefsen, Jiří Borovec, Justus Schock, Ananya Jha, Teddy Koker, Luca Di Liello, Daniel Stancl, Changsheng Quan, Maxim Grechkin, William Falcon · The Journal of Open Source Software · 2022
A main problem with reproducing machine learning publications is the variance of metric implementations across papers.A lack of standardization leads to different behavior in mechanisms such as checkpointing, learning rate schedulers or early stopping, that will influence the reported results.For example, a complex metric such as Fréchet inception distance (FID) for synthetic image quality evaluation (Heusel et al., 2017) will differ based on the specific interpolation method used.