The implementation and evaluation of a coherent memory abstraction for NUMA multiprocessors
Alan L. Cox · 1992
Most high-performance multiprocessors have a memory hierarchy in the form of caches, a distributed main memory, or some combination of both. Because of the locality of reference exhibited by typical programs, caches and distributed main memories can be used to reduce the average memory access time. To take advantage of this property, it is necessary to place threads of control and the data they access close to each other in the multiprocessor. Small bus-based multiprocessors typically have hardware that performs data placement automatically. However, Non-Uniform Memory Access (NUMA) multiprocessors require that software manage data placement. Traditionally, the responsibility for deciding data placement on NUMA multiprocessors has been left to the application programmer. In this dissertation, we describe and evaluate a software system, Coherent Memory, that automatically performs data placement on NUMA multiprocessors. A key to finding a data placement that reduces memory access time on NUMA multiprocessors is the ability to differentiate between different data sharing patterns. The thesis of this dissertation is that the operating system kernel for NUMA multiprocessors can differentiate between many different data sharing patterns at run-time and thereby select the memory access mechanism that is appropriate for data on a page-by-page basis. To evaluate this thesis, we used a suite of parallel programs running on Coherent Memory. For each program, we measured the execution time using automatic data placement by Coherent Memory and manual data placement by the programmer. We found that for programs in our application suite with a coarse-grain data sharing pattern, manual data placement outperformed Coherent Memory by less than 10%. For some of our programs, we also found that our dynamic data placement policy dramatically outperforms static data placement policies from the literature. Given a high-quality implementation, we contend that a dynamic data placement policy is preferable.