Partitioner: A Distributed HDFS Metadata Server Cluster
Ruini Xue, Lixiang Ao, Shengli Gao, Zhongyang Guan, Lupeng Lian · 2014
The Hadoop Distributed File System (HDFS) provides scalable, high throughput and fault-tolerant file service for Hadoop distributed frameworks. However, its single Name Node architecture can lead the metadata server to be a performance bottleneck and a single point of failure (SPOF). In this study, we designed and implemented Partitioner, a distributed HDFS metadata server cluster, which can unite multiple Name Nodes and provide single unique namespace for clients. To support the distributed metadata management, a novel sub tree based metadata partitioning and lookup scheme is proposed. Experiment results indicated that this method can eliminate SPOF while expanding metadata capacity, resulting in improved scalability and availability of the distributed file system.