Data wrangling at scale
Nikolay Nikolov, Michele Ciavotta, Flavio De Paoli · 2018
This paper presents a subsystem of a comprehensive platform dedicated to data transformation, linking and extension of large data sets. Furthermore, we detail and discuss both the main requirements that have led to the design and development of the platform, and the devised approach, which is a direct outcome of the requirement elicitation and discussion phase. In particular, the platform supports both design and run time aspects of the data transformation process, which is reflected in the architecture. Some initial tests have been carried out on a prototype implementation of our architecture on data sets of ~1TB featuring promising performance.