DATA AUGMENTATION IN TEXT CLASSIFICATION WITH MULTIPLE CATEGORIES
Bohdan M. Pavlyshenko, M. Stasiuk · Electronics and Information Technologies · 2024
In the modern world, the amount of text data that is being generated every day is enormous. However, because of the differences in various language usage in day-to-day life, the amount of data generated in English is much greater than for example, Ukrainian. Moreover, there are a huge amount of languages that may become extinct in the near future. Because of this, there is a request for the methods and techniques that will make it possible to preserve endangered languages and will allow us to use them effectively in the machine learning approaches. One of the developed methods for creating new data based on already existing information is called augmentation. The purpose of this article is to investigate the effect of data augmentation on the multi-class text classification task, which is performed by different transformer models: BERT, DistilBERT, ALBERT, and XLM-RoBERTa. Data for the models’ training and testing were taken from the HuggingFace. Data themselves were modified using different augmentation techniques: on the word level synonym, antonym, and contextual word embeddings augmentation were used; on the sentence level abstractive summarization and lambada augmentations were utilized. Instead of direct training and evaluation, training infrastructure, provided by the HuggingFace portal was used. Different metrics of model training efficiency were considered: learning time, the output of validation and training loss functions, accuracy, recall, f1-score, and precision. The result of this investigation allows comparing the efficiency of every observed model in multi-class text classification tasks. At the same time, the efficiency of different text augmentation was estimated. This is valuable for assessing the most corresponding transformer model in connection with augmentation to obtain the best efficiency in the classification with multiple categories. Keywords : augmentation, multi-class text classification, BERT, ALBERT, DistilBERT, XLM-RoBERTa.