Towards higher efficiency in a distributed memory storage system using data compression
Xiaoyang Yu, Songfeng Lu, Tongyang Wang, Xinfang Zhang, Shaohua Wan · International Journal of Bio-Inspired Computation · 2022
As the amount of data grows, achieving an appropriate trade-off among computation, storage and network transportation will be beneficial for a distributed memory storage system, leading to higher overall efficiency. To this end, we explore a method to achieve this trade-off by introducing data compression technology in a transparent manner. Instead of focusing on specific compressed data structures, we target block level compression for a general-purpose storage system to incorporate a wide range of existing data analysis frameworks and usage scenarios, especially with big data. A prototype is implemented and evaluated based on the memory-centric distributed storage system Alluxio to provide transparent compression and decompression during write/read operations. The extensive experiments for data with different types of compression ratio are conducted and the experimental results prove that our approach can achieve huge write/read throughput.