Cluster Interconnect Network Supporting Shared Memory
Cai Fei · Mini-micro Systems · 2007
A custom high-performance cluster network named FRAMP (Fast Remote Access and Message Passing) is designed in this paper.It not only provides fast message passing,but also gives hardware supports for shared virtual memory, including RDMA and automatic TLB updates.The FRAMP network uses exhausted cut-through switching improved from wormhole switching,slack flow controlling,source-based routing and synchronous data transfer.A custom user-level protocol is implemented,which greatly reduces the overhead of protocol and OS.The prototype of an input-queued distributed-scheduling crossbar switch is designed and implemented,which completes a round-robin scheduling in a clock cycle.The FRAMP NIC provides four data transfer modes: PIO messages, DMA messages,RDMA writes and RDMA reads.The FRAMP network achieves good performance.When no blocking,the FRAMP API latency is 5.57μs. The message bandwidth of FRAMP API is 125MBps,which achieves 94% efficiency of 32bit 33MHz PCI bus.The delay for FRAMP RDMA write is 7.15μs and the one for read is 8.00μs.The shared virtual memory system JIAJIA is modified and implemented on FRAMP network.Through application testing, it is found that FRAMP network achieves comparable performance to Myrinet and SCI and that FRAMP network has better performance when small messages have big ratio in applications.