A data optimized I/O middleware for accessing heterogeneous storage hierarchies in high performance computing and cloud environments
Ruonan Wang · UWA Profiles and Research Repository (UWA) · 2017
One of the future world's largest data generators, the Square Kilometre Array telescope (SKA), is currently approaching the construction phase. As a budget-limited project, the efficiency of the data system limits its final scope. This thesis identifies bottlenecks and potential solutions. A generic I/O framework SHORE has been designed and implemented. SHORE addresses two main points, which have not been in the focus previously. One is the recognition of the fact that one data format does not fit all requirements. The other point is the separation of user application specific data models from I/O specific and data flow optimisation techniques.