Layout-Aware Text Representations Harm Clustering Documents by Type
Catherine Finegan‐Dollak, Ashish Kumar Verma · 2020
Clustering documents by type-grouping invoices with invoices and articles with articles-is a desirable first step for organizing large collections of document scans.Humans approaching this task use both the semantics of the text and the document layout to assist in grouping like documents.Lay-outLM (Xu et al., 2019), a layout-aware transformer built on top of BERT with state-of-theart performance on document-type classification, could reasonably be expected to outperform regular BERT (Devlin et al., 2018) for document-type clustering.However, we find experimentally that BERT significantly outperforms LayoutLM on this task (p < 0.001).We analyze clusters to show where layout awareness is an asset and where it is a liability.