Distributed and High Performance Big-File Cloud Storage Based On Key-Value Store
Thành Trung Nguyễn, Minh Hieu Nguyen · The International journal of networked and distributed computing · 2016
This research proposes a new Big File Cloud (BFC) with its architecture and algorithms to solve difficult problems of cloud-based storage using the advantages of key-value stores.There are many problems when designing an efficient storage engine for cloud-based storage systems with strict requirements such as big-file processing, lightweight meta-data, low latency, parallel I/O, deduplication, distributed, high scalability.Keyvalue stores have many advantages and outperform traditional relational database in storing data for heavy load systems.This paper contributes a low-complicated, fixed-size meta-data design, which supports fast and highly-concurrent, distributed file I/O, several algorithms for resumable upload, download and simple data deduplication method for static data.This research applies the advantages of ZDB -an in-house key-value store which was optimized with auto-increment integer keys for solving big-file storage problems efficiently.The results can be used for building scalable distributed data cloud storage that support big-files with sizes up to several terabytes.