Characterizing Text Datasets with Psycholinguistic Features
Marcio Monteiro, Charu Karakkaparambil James, Marius Kloft, Sophie Fellenz · 2024
Fine-tuning pretrained language models on task-specific data is a common practice in Natural Language Processing (NLP) applications.However, the number of pretrained models available to choose from can be very large, and it remains unclear how to select the optimal model without spending considerable amounts of computational resources, especially for the text domain.To address this problem, we introduce PsyMatrix, a novel framework designed to efficiently characterize text datasets.Psy-Matrix evaluates multiple dimensions of text and discourse, producing interpretable, lowdimensional embeddings.Our framework has been tested using a meta-dataset repository that includes the performance of 24 pretrained large language models fine-tuned across 146 classification datasets.Using the proposed embeddings, we successfully developed a metalearning system capable of recommending the most effective pretrained models (optimal and near-optimal) for fine-tuning on new datasets.