Document Alignment based on Overlapping Fixed-Length Segments
Xiaotian Wang, Takehito Utsuro, Masaaki Nagata · 2024
Acquiring large-scale parallel corpora is crucial for NLP tasks such as Neural Machine Translation, and web crawling has become a popular methodology for this purpose.Previous studies have been conducted based on sentencebased segmentation (SBS) when aligning documents in various languages which are obtained through web crawling.Among them, the TK-PERT method (Thompson and Koehn, 2020) achieved state-of-the-art results and addressed the boilerplate text in web crawling data well through a down-weighting approach.However, there remains a problem with how to handle long-text encoding better.Thus, we introduce the strategy of Overlapping Fixed-Length Segmentation (OFLS) in place of SBS, and observe a pronounced enhancement when performing the same approach for document alignment.In this paper, we compare the SBS and OFLS using three previous methods, Mean-Pool, TK-PERT (Thompson and Koehn, 2020), and Optimal Transport (Clark et al., 2019;El-Kishky and Guzmán, 2020), on the WMT16 document alignment shared task for French-English, as well as on our self-established Japanese-English dataset MnRN.As a result, for the WMT16 task, various SBS based methods showed an increase in recall by 1% to 10% after reproduction with OFLS.For MnRN data, OFLS demonstrated notable accuracy improvements and exhibited faster document embedding speed.