Improving Communication Performance on InfiniBand by Using Efficient Data Placement Strategies
Robert Rex, Frank Mietke, Wolfgang Rehm, C. Raisch, Hoang-Nam Nguyen · 2006
Despite using high-speed network interconnection systems like InfiniBand, the communication overhead for parallel applications is still high. In this paper we show, how such costs can be reduced by choosing appropriate data placement strategies. For large buffers, we propose a transparent placing in hugepages as it can dramatically decrease memory registration overhead and may increase network bandwidth. Thus, we developed a new library that can be preloaded for applications at load time and cares about drawbacks of using hugepages. So we believe that it is the most suitable one in the HPC area for Linux today. But we do not only refer to large buffers as small communication buffers also play a significant role for application behaviour. We show that transfer latencies vary, depending on data placement. All current communication library implementations for InfiniBand do not utilize scatter-gather lists for send and receive operations, but we show that this feature can have a positive impact on latency for small buffers and data aggregation can perform better. Our results show that communication performance of applications may improve more than 10 % using the presented improvements