Large Scale Parallel Document Mining for Machine Translation
Jakob Uszkoreit, Jay M. Ponte, Ashok C. Popat, Moshe Dubiner · 2010
A distributed system is described that reliably mines parallel text from large corpora. The approach can be regarded as cross-language near-duplicate detection, enabled by an initial, low-quality batch translation. In contrast to other approaches which require specialized metadata, the system uses only the textual content of the documents. Results are presented for a corpus of over two billion web pages and for a large collection of digitized public-domain books. 1