The Optimistic Direct Access File System: Design and Network Interface Support
Kostas Magoutis · 2002
Abstract | The emergence of commercially-available net-work interface controllers (NICs) with remote direct mem-ory access (RDMA) capability and the prospect of their tighter integration with the host memory system moti-vate the design of distributed systems based on an RDMA paradigm. A recent example is the Direct Access File Sys-tem (DAFS). DAFS clients communicate requests to servers using lightweight Remote Procedure Calls (RPC) based on low-overhead message passing. Data can be transmitted ei-ther in-line with the messages, or via server-initiated RDMA independent of the messages. DAFS clients do not initiate RDMA, despite the potentially lower latency of this mech-anism for short transfers. With current NICs, servers that export their entire le cache would have to resort to ex-cessive page wiring to guarantee success of client-initiated RDMA operations. In this paper we present the design of the Optimistic Di-rect Access File System, a distributed lesystem that en-hances DAFS by enabling clients to directly access remote memory pages of cached les exported by servers. Using our proposed NIC support, exported pages are not per-manently wired in physical memory except during RDMA operations referencing them. Client RDMA attempts for pages that are no longer memory-resident result in excep-tions thrown by the RDMA target (server) and caught by the initiator (client), prompting a switch to an alternative access method. We present simulation results showing that optimistic client-initiated RDMA seldom fails when client working sets t in server physical memory. Microbench-marks with a prototype RDMA-capable NIC designed to support the Optimistic Direct Access File System show that client-initiated RDMA can achieve lower le access latency when compared to le access using RPC. The applicability of optimistic client-initiated RDMA extends to other domains, such as Distributed Shared Memory systems. I.