Improving Access to HDFS using NVMeoF
Daegyu Han, Beomseok Nam · 2019
In this extended abstract, we discuss how to improve access to HDFS by employing NVMe over Fabrics (NVMeoF), which has emerged as a new communication protocol between a host and a storage system. To address the legacy shortcomings of HDFS, we explore the opportunity of enabling all-to-all connections between NVMe SSDs and DataNodes rather than dedicating each NVMe to a single DataNode. Our experimental study shows that we can achieve up to 2.55 times higher I/O throughput than legacy HDFS by making HDFS leverage NVMeoF for remote data access.