Distributed file system for rewriting Big Data files using a local-write protocol
Erico Correia Da Silva, Líria Matsumoto Sato, Edson Toshimi Midorikawa · 2021 IEEE International Conference on Big Data (Big Data) · 2021
With the exponential volume growth of the data available for scientific and commercial use, more and more Big Data technologies are gaining focus and importance. Directly related to the efficiency of these techniques is the distributed file system used for data persistence, generally based on low-cost computer clusters. However, the environments used today for Big Data are based on file systems restricted to the WORM pattern (write once, read many) lacking POSIX compatibility. This work uses distributed lock management techniques to create a file system that allows random writing for both HPC and Big Data tools. A local write protocol is implemented to leverage the use of local copies of the data during the write process. Experiments were carried out to evaluate the performance of the proposed write protocol and the scalability of the developed file system. From the experimental results, it is possible to conclude that the achieved performance and scalability improvements were obtained by eliminating limitations imposed by HDFS and leveraging local writes.