The case for reconfigurable I/O channels
Steven G. Smith, Anil Madhavapeddy, Christopher Smowton, Malte Schwarzkopf, Richard Mortier, Robert M. Watson, Steven J. Hand · 2012
Datacenter environments are increasingly layered, with multicore parallelism, OS virtualisation and NUMA memory all introducing variable latency and throughput for data transmission. For a pro-grammer deploying applications in such a shifting environment, it is unclear how best to use venerable interfaces such as the sockets layer. Kernel hackers realise there is some performance hit to all the software layering, but quantitative figures are hard to find. This is a position paper of two rather different halves. We first seek to understand just how big the impact of NUMA layouts and OS virtualisation have been on I/O performance. To do this, we implemented a variety of IPC mechanisms (from TCP sockets to shared memory) and benchmarked them under modern multicore hardware and Xen. We discovered a large variance in throughput and latency under different scheduling conditions (over an order of magnitude in some cases), and also some rather inexplicable results which point to the extreme difficulty of predicting cross-layer performance. In the second half, we describe the early design of a system which aims to overcome these multiplexed I/O scheduling issues. It provides an efficient, zero-copy data transmission interface that automates the selection of the underlying transport, and the fa-cility to dynamically reconfigure transports as system conditions change. Finally, we discuss the implications of extending the OS with explicit I/O flow tracking—eliminating contention, transpar-ent transport-level security and an upgrade path to multi-path TCP. 1.