Analysis of GraphSum’s Attention Weights to Improve the Explainability of Multi-Document Summarization
M. Lautaro Hickmann, Fabian Wurzberger, Megi Hoxhalli, Arne Lochner, Jessica Töllich, Ansgar Scherp · 2021
Modern multi-document summarization (MDS) methods are based on transformer architectures. They generate state of the art summaries, but lack explainability. To overcome this, we analyze the attention weights of a graph-based MDS such as GraphSum. We compare GraphSum’s performance utilizing different textual units, i. e., sentences versus paragraphs, on two MDS benchmark datasets, namely WikiSum and MultiNews. Our experiments show that paragraph-level representations provide the best summarization performance. Subsequently, we focus on analyzing the paragraph-level attention weights of GraphSum’s multi-heads and decoding layers. Furthermore, we examine source origin information via text similarity over a ROUGE-based reference metric. We observe a high correlation between the attention weights and our reference metric, especially on the later decoding layers. Finally, we investigate if the generated summaries follow a pattern of positional bias. Our results show that there is a high correlation between the position of paragraphs in the input documents and the content provided for the generated summary.