A scalable data distributing method supporting replication and weight
Wenhan Zhan, Hancong Duan, Lin Li, Xiaowen Nie, Cheng Chen · International Conference on Computing Technology and Information Management · 2012
To design a large-scale distributed storage system, availability, scalability, user experience, and resource utilization are all under consideration. Known approaches for distributing and locating a huge amount of objects in such systems cannot meet all the requirements. In this paper, we propose a new method which uses three-level mapping to locate objects. With this method, objects can be distributed among a large number of storage devices with high scalability. Clients can access objects in parallel without consulting a central server each time, and the cost to locate an object is one step in general. Moreover, replication and weighted allocation can also be supported, both of which are needed to permit systems to efficiently grow while accommodating new technology. Theoretical proofs and experiments show that our method is comparable to other methods, and can meet all the requirements to achieve an advanced large-scale distributed storage system.