Data Selection Driven by Item Difficulty: On Investigating Data Efficient Practice for Hyperparameter Search
Gustavo Rodrigues dos Reis, Adrian Mos, Mario Cortes Cornax, Cyril Labbé · 2024
Foundation Models shift the interest to adapting models instead of creating proprietary models from scratch. Despite this change, performing hyperparameter optimization (HPO) is still needed. Users adapting systems powered by those models on proprietary data should not considerably increase the overall resource footprint with extensive hyperparameter search. Given that this footprint is also proportional to the data used in HPO, we aim to investigate how a user can effectively reduce the amount of data used, leveraging the deep learning model's empirical facility to output the expected correct result for an item in the dataset.