BERT meets Cranfield: Uncovering the Properties of Full Ranking on Fully Labeled Data
Negin Ghasemi, Djoerd Hiemstra · 2021
Recently, various information retrieval models have been proposed based on pre-trained BERT models, achieving outstanding performance.The majority of such models have been tested on data collections with partial relevance labels, where various potentially relevant documents have not been exposed to the annotators.Therefore, evaluating BERTbased rankers may lead to biased and unfair evaluation results, simply because a relevant document has not been exposed to the annotators while creating the collection.In our work, we aim to better understand a BERTbased ranker's strengths compared to a BERTbased re-ranker and the initial ranker.To this aim, we investigate BERT-based rankers performance on the Cranfield collection, which comes with full relevance judgment on all documents in the collection.Our results demonstrate the BERT-based full ranker's effectiveness, as opposed to the BERT-based re-ranker and BM25.Also, analysis shows that there are documents that the BERT-based full-ranker finds that were not found by the initial ranker.