Data Distribution Algorithms in Distributed File Systems
Xinyu Zuo, Ning Hu, Peng Zhang, Zelan Yuan, Yao Zhao · 2024
With the continuous growth of massive data, it becomes more and more important to effectively manage and store data in distributed file system. Data distribution is one of the key problems. This paper introduces the concept of distributed file system and the importance of data distribution, and studies the data distribution algorithm, which is divided into static and dynamic two categories. Static data data distribution methods are usually classified according to the location or attributes of data, including spatial partition, data attribute and data access mode. Dynamic data distribution methods manage data based on real-time load conditions to achieve optimal system performance, including dynamic load balancing, traffic aware data placement and preference learning. In addition, the paper also discusses the system architecture and data distribution mode of several typical distributed file systems such as HDFS, Ceph and GlusterFS. Finally, the paper summarizes the data distribution algorithm of distributed file system and puts forward the understanding of its future development.