OPEN: Passive Network Performance Estimation for Data-intensive Applications
Jinoh Kim, Abhishek Chandra, Jon Weissman · University of Minnesota Digital Conservancy (University of Minnesota) · 2008
Distributed computing applications are increasingly utilizing distributed data sources. However, the unpredictable cost of data access in large distributed systems can lead to severe performance bottlenecks. Providing predictability in data access is essential to accommodate the large set of newly emerging data-intensive applications. Accurate estimation of data access costs are crucial to meeting the performance goals of such applications. In this paper, we present a framework called OPEN, Overlay Passive Estimation of Network performance, which provides application-perspective network performance estimation based on past downloading measurements. Unlike previous passive estimation work, OPEN enables nodes to share measures in a scalable system-wide manner without any topological constraints. It uses passive estimation and gossip-based dissemination of measurements to achieve scalability. Using data download traces collected for 10 months in Planet-Lab, we show that our framework is widely applicable to selection problems common in distributed computing applications. Results from our simulation study show that OPEN significantly outperforms selection techniques based on statistical pairwise estimations as well as random and latency-based selections in diverse experimental settings.