iDPL: A scalable and flexible inter-continental testbed for data placement research and experiment
Guang Hua Wei, Hailong Yang, Zhongzhi Luan, Depei Qian · 2017
In this paper, we propose the China-US international data placement laboratory (iDPL) based on an inter-continental testbed for data placement research. iDPL is able to support various data placement research due to its scalability and flexibility in deploying the experiments in the real network environment. The core design of iDPL leverages reliable workflow management and lightweight I/O protocol to allow complex experiment setup and on-the-fly experiment deployment. It is also extensible to plugin different network profiling tools such as iperf. We expect the powerful measurement capability of iDPL promotes research study on the intelligent data placement policies which adapt to the uncertainty of the wide-area network and guarantee the quality of service (QoS) of the big data applications. As a case study, we setup a set of data placement experiments to measure the end-to-end network performance constantly among several sites between China and US using different data placement tools. The experiments have been running for more than one year, and its measurement data is public available (http://mickey.buaa.edu.cn:8080/). We believe the measurement data is valuable for both network and big data researchers to understand the performance disparity between the raw network and the actual data placement, which provides useful insights to design big data applications with performance awareness. We encourage more researchers to deploy their own data placement experiments on iDPL, expediting the research direction of intelligent data placement with real network environment.