Storage hierarchy management for scientific computing
Ethan L. Miller · 1996
Leo Miller Scientific computation has always been one of the driving forces behind the design of computer systems. As a result, many advances in CPU architecture were first developed for high-speed supercomputer systems, keeping them among the fastest computers in the world. However, little research has been done in storing the vast quantities of data that scientists manipulate on these powerful computers. This thesis first characterizes scientists ’ usage of a multi-terabyte tertiary storage system attached to a highspeed computer. The analysis finds that the number of files and average file size have both increased by several orders of magnitude since 1980. The study also finds that integration of tertiary storage with secondary storage is critical. Many of the accesses to files stored on tape could have easily been avoided had scientists seen a unified view of the mass storage hierarchy instead of the two separate views of the system studied. This finding was a major motivation of the design of the RAMA file system. The remainder of the thesis describes the design and simulation of a massively parallel processor (MPP) file system that is simple, easy to use, and integrates well with tertiary storage. MPPs are increasingly commonly used for scientific computation, yet their file systems require great attention to detail to get acceptable