CorporAl: a Method and Tool for Handling Overlapping Parallel Corpora
Mark Fishel, Heiki-Jaan Kaalep · The Prague Bulletin of Mathematical Linguistics · 2010
CorporAl: a Method and Tool for Handling Overlapping Parallel Corpora This work introduces a method and tool for handling overlapping parallel corpora — i.e. corpora that are based on the same source material. The method is insensitive to minor changes in the text, different segmentation levels of the corpora and omitted material from either corpora. The aim is to detect matching sentence pairs and either produce combinations of the overlapping corpora or compare them and assess their quality in comparison to each other. The introduced tool enables the user to define the desired behavior when combining corpora pairs, resulting in pure comparison, maximum-size or maximum-quality versions of the combinations. We test the tool on two cases of overlapping parallel corpora and five language pairs. We also evaluate the impact of using the method on two translation systems — a phrase-based and a parsing-based one.