U-Net Encapsulated Transformer for Reducing Dimensionality in Training Large Language Models
Marvin John Ignacio, Yong-Guk Kim, Hulin Jin, Seunghee Yu · ACM Transactions on Intelligent Systems and Technology · 2025
Training language models from scratch presents a critical challenge in Natural Language Processing (NLP), primarily due to the computational demands of pre-trained Large Language Models, which are predominantly trained on English corpora using extensive resources. While offering viable solutions, existing alternatives still rely heavily on high-performance hardware. This work introduces a different approach to reducing the algorithmic complexity of Transformer-based architectures through the U-Net Encapsulated Transformer (UET), which applies dimensionality reduction to token embeddings. The UET architecture enables the development of language models with significantly reduced parameter sizes for a given set of hyperparameters. Alternatively, it allows researchers to design models of comparable size but with a substantially greater number of Transformer blocks, enhancing model depth and potential capacity. This study also outlines practical methodologies for training language models in resource-constrained environments. Experimental results illustrate the potential of the UET architecture in achieving reasonable performance under resource-constrained conditions, highlighting its promise as an accessible alternative for language model development. This work could broaden the accessibility of NLP research, empowering researchers with hardware constraints to contribute to language model development.