OpenWebSearch.eu - Building an Open Web Index on EuroHPC JU Infrastructures
Michael Granitzer, Mohamad Hayek, Sebastian Heineking, Gijs Hendriksen, Martin Golasowski, Michael Dinzinger, Saber Zerhoudi · Procedia Computer Science · 2025
The OpenWebSearch.eu project aims to develop an Open Web Index (OWI), an openly accessible data structure that supports the creation of web search engines. Building such an index requires a data- and compute-intensive pipeline for cleaning, preprocessing, enriching and indexing large amounts of web data. Beyond search, the availability of clean and preprocessed web data is also crucial for fields like web analytics and generative AI. This paper presents our approach to constructing the OWI using High-Performance Computing (HPC) resources from both EuroHPC JU and non-EuroHPC JU data centers. We contribute in two main areas: first, by detailing the development of pre-processing and indexing pipelines embedded in HPC workflows; and second, by describing the iRODS-based federated storage infrastructure and the LEXIS 1 platform that will manage the cross-data centre workflows and facilitate the publication of the OWI as daily datasets. During the alpha phase, from October 2023 to April 2024, we processed approximately 76 TB of web data, encompassing over 2 billion URLs. By addressing the challenges of large-scale web data processing and retrieval, this work lays the foundation for an innovative, competitive, and transparent web search ecosystem, while also supporting the development of European generative AI solutions.