An Adaptive File Distribution Algorithm for Wide Area Network
Takashi Hoshino, Kenjiro Taura, Takashi Chikayama · 2001
Abstra t. This paper des ribes a data distribution algorithm suitable for opying large les to many nodes in multiple lusters in wide-area networks. It is a self-organizing algorithm that a hieves pipeline transfers, fault toleran e, s alability, and an e ient route sele tion. It works in the presen e of today's typi al network restri tions su h as rewalls and Network Address Translations, making it suitable in wide-area setting. Experimental results indi ate our algorithm is able to automati ally build a transfer route lose to the optimal. Propagation of a 300MB le from one root node to over 150 nodes takes about 1.5 times as long as the best time obtained by the manually optimized transfer route. Key words. Self-stabilizing distributed algorithm, fault toleran e, s alability, wide-area network 1. Introdu tion. This paper des ribes a pra ti al algorithm for opying large data (typi ally in a le) from a sour e node(s) to many destination nodes in parallel. We seek a s alable solution suitable both within a luster and a ross many lusters in wide-area. By suitable within a luster, we mean that it fully utilizes the available bandwidth of LAN / luster inter onne t. For example, assuming 32 nodes are onne ted via a su iently high-throughput swit h, it should be able to opy a single large le to the 32 nodes in not mu h more than the time it takes to opy the le to a single node. Su h an algorithm must at least perform many one-to-one transfers in parallel. By suitable in wide-area, we mean it makes a good hoi e in sele ting transfer routes. If many nodes in a luster retrieve data from another luster, a link a ross the two easily saturates.