Optimizing parallel input/output using adaptive file system policies
Daniel A. Reed, Christopher L. Elford · 1998
Recent research in techniques for optimizing parallel input/output has shown that matching available file system policies to application access patterns and resource requirements can produce substantial performance improvements. However, parallel file system policy availability varies significantly across computation platforms. Similarly, parallel scientific access patterns and resource requirements vary not only across applications but also among phases within an application. Unfortunately, the potential range of policies and access patterns make an exhaustive search of the policy space for optimal combinations prohibitively expensive. We propose a two phase optimization strategy that first applies factor analysis to identify the most effective policy combinations on a given platform. Second, at run time, it performs closed loop control to refine policies. Rather than requiring potentially millions of experiments to identify an effective 'optimal' policy combination, factor analysis uses a reasonable number of experiments to identify an effective, though not provably optimal, policy. Having identified effective policies on a given platform, we automatically monitor access patterns and system performance at application run time and refine file system policy parameters as bottlenecks become evident. We have implemented this approach via several extensions to the Portable Parallel File System (PPFS) testbed. Sensor metrics that summarize dynamic access pattern information and file system performance effectively guide adaptive policy selection, affording significant performance improvements over less adaptive optimization strategies. Our results with both parallel access pattern benchmarks and parallel scientific applications demonstrate the efficacy of our approach.