MemepiC: Towards a Unified In-Memory Big Data Management System
Qingchao Cai, Hao Zhang, Wentian Guo, Gang Chen, Beng Chin Ooi, Kian‐Lee Tan, Weng‐Fai Wong · IEEE Transactions on Big Data · 2018
In-memory data management systems have recently gained a lot of attraction due to cheaper and faster DRAM and other hardware advancement. However, these systems are either pure storage systems with online data query service, or just offline batch processing systems with data analytics functionality. Heavy data movement (e.g., data loading) occurs in order to analyze the data. In this paper, we propose an innovative in-memory data management system-MemepiC, which unifies both online data query and data analytics functionality, allowing low-latency storage service and efficient in-situ data analytics. We also explore the emerging RDMA technique in the context of in-memory data management systems, by designing an RDMA-based communication protocol for message delivery inside MemepiC, and proposing to overlap execution and RDMA communication. Extensive experiments are conducted to show the superior performance of MemepiC in terms of both the storage and the data analytics services, compared against other in-memory systems.