Evaluating Performance of Transformer Models for Dialogue Summarization: A Comparison of T5-Base, T5-Small, and BART-Base
Delta Setiyarini, Teguh Bharata Adji, Indriana Hidayah · 2024
Dialogue summarization transforms multispeaker dialogues into coherent and concise summaries. Previous research has primarily used large-parameter models like BART-Large and Pegasus. These models require significant computational resources, making them less accessible. Smaller models like T5-Small have not been thoroughly evaluated for dialogue summarization tasks. This research addresses that gap by comparing the performance and efficiency of T5-Base, T5Small, and BART-Base on the DialogSum dataset. The models were evaluated using ROUGE metrics, including ROUGE-1, ROUGE-2, and ROUGE-L. T5-Base consistently outperformed the other models, achieving the highest ROUGE-1 score of 0.4732. Despite its smaller size, T5-Small had a competitive performance with a ROUGE-1 score of 0.4287. T5-Small also outperformed BART-Base, which had a score of 0.4099. T5Small had faster training times compared to T5-Base. This makes T5-Small more suitable for resource-constrained environments. These results show that T5-Small has potential for abstractive summarization tasks. Future research should focus on optimizing model architectures and minimizing training loss to enhance performance further.