SCALABLE CATALOG INFRASTRUCTURE FOR MANAGING ACCESS COSTS AND SOURCE SELECTION IN WIDE AREA NETWORKS
Vladimir Zadorozhny, Louiqa Raschid, Avigdor Gal · International Journal of Cooperative Information Systems · 2008
A WAN environment, such as the Internet, connects a federation of hundreds of servers with tens of thousands of clients, which poses a substantial scalability challenge. Clients may choose among sources that vary in both their content and quality as well as in their access latencies. At the same time, Internet accessible data sources exhibit transient behavior; the unpredictable behavior of a dynamic WAN results in a wide variability in access cost (end-to-end latency). This motivates a need for a source selection strategy that requires maintaining access cost distributions (latency profiles) for each client/server pair. However, in the presence of hundreds of servers and thousands of clients, managing latency profiles cannot scale. We present a scalable methodology to manage latency profiles that use non-random associations between client/server pairs. Such non-random associations may be identified by topology-independent measures such as correlation and mutual information. We propose a Catalog infrastructure that implements our methodology and utilize non-randomly associated latency profiles to estimate access cost distribution for client/server pairs. We perform an extensive experimental study demonstrating feasibility and efficiency of our approach.