Deep scientific computing requires deep data
William Kramer, Arie Shoshani, D. Agarwal, Brent R. Draney, G. Jin, Gregory F. Butler, John Hules · IBM Journal of Research and Development · 2004
Increasingly, scientific advances require the fusion of large amounts of complex data with extraordinary amounts of computational power. The problems of deep science demand deep computing and deep storage resources. In addition to teraflop-range computing engines with their own local storage, facilities must provide large data repositories of the order of 10–100 petabytes, and networking to allow the movement of multi-terabyte files in a timely and secure manner. This paper examines such problems and identifies associated challenges. The paper discusses some of the storage systems and data management methods that are needed for computing facilities to address the challenges and describes some ongoing improvements.