I/O friendly data parallelization for spatial computation
Baoqiang Yan, Philip J. Rhodes · 2008
Due to disk and network latencies, I/O performance remains a major bottleneck for HPC on large datasets. As an important I/O optimization technique, prefetching and caching are widely employed in modern file systems to speed up data access. However, they are optimized for sequential locality and not usually effective for volumetric scientific data retrieval because spatial locality in query does not correspond to proximity in linear storage. Also, a prefetching policy must not only be efficient, having low overhead, but must also be effective, choosing the correct blocks to prefetch. The information about the future access pattern can help improve the effectiveness of prefetching and cache hit rate. Profile or hint based prefetching assumes that the complete access pattern is not available in advance and relies on certain prediction models to guess what to preload, which is error prone. In fact, many applications such as FFT and ray casting have regular access patterns that are known a priori and thus can be exploited to improve cache performance.