Scalable Construction of High-Quality Web Corpora
Chris Biemann, Felix Bildhauer, Stefan Evert, Dirk Goldhahn, Uwe Quasthoff, Roland Schäfer, Johannes Simon, Leonard Swiezinski, Torsten Zesch · LDV-Forum/Journal for language technology and computational linguistics · 2013
In this article, we give an overview about the necessary steps to construct high-quality corpora from web texts.We first focus on web crawling and the pros and cons of the existing crawling strategies.Then, we describe how the crawled data can be linguistically pre-processed in a parallelized way that allows the processing of web-scale input data.As we are working with web data, controlling the quality of the resulting corpus is an important issue, which we address by showing how corpus statistics and a linguistic evaluation can be used to assess the quality of corpora.Finally, we show how the availability of extremely large, high-quality corpora opens up new directions for research in various fields of linguistics, computational linguistics, and natural language processing.