Dawning Nebula Distributed File System HVFS:for Large Scale Small File Access
Jin Xiong · Journal of Chinese Computer Systems · 2012
The emerging popular Internet applications such as online photo,audio,video and micro blogging services exhibit very different data access and storage requirements from traditional applications.Large number of small data are generated,analyzed,and returned every second in data center.These applications requires challenging performance for highly concurrent both high throughput and low latency access to tiny files.In this paper,we propose HVFS,a novel distributed file system built over distributed tabular storage,to manage billions of small files and support highly concurrent accesses.HVFS uses extendible hash to index metadata,log-structured storage format and columnar storage to exploit temporal and spatial locality.We present the design and implementation of HVFS.Our evaluation results demonstrate that the core tabular storage of HVFS can serve more than 100,000/240,000 aggregated data read/write requests per second on 82 nodes(1KB),and FUSE implementation can serve more than 180,000 aggregated file creations per second on 32 nodes.