Static and Dynamic Contextual Embedding for AutoML in Text Classification Tasks
Parisa Safikhani, David Broneske · 2025
AutoML streamlines NLP model development by automating tasks like model selection and hyperparameter tuning. However, integrating language models like BERT into AutoML poses challenges in computational efficiency and adaptability. In this paper, we introduce two novel approaches, StaticBERT Auto-PyTorch, and DynamicBERT Auto-PyTorch which incorporate static and dynamically fine-tuned BERT embeddings into the Auto-PyTorch framework for binary and multi-class text classification, respectively. StaticBERT Auto-PyTorch leverages pre-fine-tuned BERT embeddings to enhance performance and reduce memory usage in binary tasks. Conversely, DynamicBERT Auto-PyTorch fine-tunes BERT embeddings for each dataset, improving adaptability and meta-learning capabilities for both binary and multi-class classifications, albeit with longer processing times. Extensive experiments on diverse English and German classification datasets demonstrate that our methods significantly outperform traditional AutoML approaches in both prediction performance and computational efficiency. These findings underscore the pivotal role of contextual embeddings in advancing AutoML, providing robust and scalable solutions for complex NLP classification challenges. The choice between StaticBERT and DynamicBERT depends on task requirements, balancing the trade-off between processing speed and adaptability to diverse linguistic contexts.