Inference-Based No-Learning Approach on Pre-Trained BERT Model Retrieval
Huu-Long Pham, Ryota Mibayashi, Takehiro Yamamoto, Makoto P. Kato, Yusuke Yamamoto, Yoshiyuki Shoji, Hiroaki Ohshima · 2024
In recent years, the practice of leveraging pre-trained machine learning models for specific tasks has gained traction. Instead of training models from the ground up, it is now common to fine-tune existing pre-trained models. However, users have the responsibility to select a pre-trained model that is suitable with their task-a challenge given the number of pre-trained models available. Conventionally, the suitability of a pre-trained model for a task is ascertained through fine-tuning, which is costly in term of time and computational resources. This research introduces an efficient retrieval method for BERT pre-trained models in document classification tasks. Our contributions are as follows: (i) We defined the problem of pre-trained model retrieval; (ii) We developed a benchmark dataset by fine-tuning and evaluating twenty distinct pre-trained BERT models across seventeen document classification tasks; (iii) We propose a method to retrieve suitable pre-trained BERT models without actual fine-tuning.