Migratory File Services for Scientific Applications
John Bent, Andrea Carol Arpaci-Dusseau, Remzi H. Arpaci-Dusseau, Miron Livny · 2002
We present the design and implementation of a migra-tory file service (MFS) for scientific applications on the Grid. An MFS is both easy to use and delivers high per-formance: two important properties that traditional Grid-based storage solutions are lacking. The key responsi-bility of an MFS is to dynamically create the user’s ex-pected file system environment on a remote cluster where the user’s jobs are running. We explore these concepts in a prototype MFS called the Hawk File Service. HFS transparently redirects appli-cation I/O to a set of migratory proxies that are located near the user’s jobs; thus, the jobs can execute as if in their home environment. HFS also intelligently manages the set of proxies to account for the working-set size and failure characteristics of the cluster; thus, the jobs execute with faster access to their files and the overall through-put of the system is improved. Through a series of mi-crobenchmarks and an application study, we demonstrate the effectiveness of the migratory file service paradigm. 1