Automatic Text Summarization-Based Transformers Architecture
Alaa Ahmed Al-Banna, Abeer Khalid Al-Mashhadany · 2023
Summarization is the process of condensing a part of the text into a shorter version, decreasing the size of the original text while keeping the central informative and content significance. Because manual text summarizing is time-consuming and typically arduous, its automation is increasing in popularity, thus serving as a powerful stimulus for academic study. This work builds a model for an abstract text summarizer for the English language using the state-of-the-art “Transformer” model and self-attention mechanism on dataset SciTLDR of 5.4K. This dataset size cannot train the model in traditional deep learning methods such as RNN and LSTM. Also, it is challenging to train in machine learning as the number of samples is insufficient, and the model may suffer from under-fitting. The results of the Transformers model showed a good performance in the dataset size. The human evaluation results showed good performance in terms of Grammaticality, language cohesion, and non-repetition of information. This paper proposes an iterative data augmentation approach and an attention mechanism process in parallel; this is particularly helpful for datasets suffering from low resource conditions to get a better result.