LODOP - Multi-Query Optimization for Linked Data Profiling Queries.
Benedikt Forchhammer, Anja Jentzsch, Felix Naumann · 2014
Abstract. The Web of Data contains a large number of different, openly-available datasets. In order to effectively integrate them into existing applications, meta information on statistical and structural properties is needed. Examples include information about cardinalities, value pat-terns, or co-occurring properties. For Linked Datasets such information is currently very limited or not available at all. Data profiling techniques are needed to compute respective statistics and meta information. How-ever, current state of the art approaches can either not be applied to Linked Data, or exhibit considerable performance problems. We present Lodop, a framework for computing, optimizing, and bench-marking data profiling techniques based on MapReduce with Apache Pig. We implemented 15 of the most important data profiling tasks, opti-mized their simultaneous execution, and evaluate them with four typical datasets from the Web of Data. Our optimizations focus on reducing the amount of MapReduce jobs and minimizing the communication overhead between multiple jobs. Our evaluation shows the significant potential in optimizing the runtime costs for Linked Data profiling. 1