Foundation Models and Fine-Tuning: A Benchmark for Out of Distribution Detection
Francesco Cappio Borlino, Lorenzo Li Lu, Tatiana Tommasi · IEEE Access · 2024
The rise of foundation models is pushing Computer Vision research towards a paradigm shift, in the wake of what already happened in the Natural Language Processing field. These models, trained at scale on huge data collections, provide high-quality representations that generalize well enough to be applied directly to downstream tasks, often outperforming task-specific models. The Out Of Distribution (OOD) detection problem, which involves the ability to recognize when test samples come from a previously unseen semantic category, represents one of the research fields in which this paradigm shift could have the greatest impact. However, existing testbeds are limited in scale and scope and get easily saturated when adopting foundation-based pretrainings. With this work, we introduce a new benchmark covering realistic yet harder OOD detection tasks to properly assess the performance of large pretrained models.We design an experimental framework to analyze specific choices in the model learning and use (which dataset, pretraining objective, OOD scoring function) and extensively evaluate the comparison to standard approaches that leverage a training phase on the available In Distribution (ID) data. The results reveal the real performances that are unlocked by exploiting foundation models in this context, and in which cases task-specific finetuning still show to be the best choice.