RIDIRE-CPI: an Open Source Crawling and Processing Infrastructure for Supervised Web-Corpora Building
Alessandro Panunzi, Marco Fabbri, Massimo Moneglia, Lorenzo Gregori, Samuele Paladini · 2012
This paper introduces the RIDIRE-CPI, an open source tool for the building of web corpora with a specific design through a targeted crawling strategy.The tool has been developed within the RIDIRE Project, which aims at creating a 2 billion word balanced web corpus for Italian.RIDIRE-CPI architecture integrates existing open source tools as well as modules developed specifically within the RIDIRE project.It consists of various components: a robust crawler (Heritrix), a user friendly web interface, several conversion and cleaning tools, an anti-duplicate filter, a language guesser, and a PoS tagger.The RIDIRE-CPI user-friendly interface is specifically intended for allowing collaborative work performance by users with low skills in web technology and text processing.Moreover, RIDIRE-CPI integrates a validation interface dedicated to the evaluation of the targeted crawling.Through the content selection, metadata assignment, and validation procedures, the RIDIRE-CPI allows the gathering of linguistic data with a supervised strategy that leads to a higher level of control of the corpus contents.The modular architecture of the infrastructure and its open-source distribution will assure the reusability of the tool for other corpus building initiatives.