A distributed shared buffer space for data-intensive applications
Renaud Lachaize, J.S. Hansen · 2005
Efficient memory allocation and data transfer for cluster-based data-intensive applications is a difficult task. Both changes in cluster interconnects and application workloads usually require timing of the application and network code. We propose separating control and data transfer traffic by accessing data through a DSM-like cluster-wide shared buffer space and only including buffer references in the control messages. Using a generic API for accessing buffers allows for tuning data transfer without changing the application code. A prototype, implemented in the context of a distributed storage system, has been validated with several networking technologies, showing that such a framework can combine performance and flexibility.