Managing and serving a multiterabyte data set at the Fermilab DO experiment
Lee Lueking · 2002
The DO experiment at Fermilab is accumulating data from the electronic detection of collisions between protons and anti-protons. The presentation describes the data structure, data cataloging and serving of the multiterabyte data set to a user community. The current data consists of over 85 terabytes stored in a hierarchy of data sets with various latencies and frequencies of use. The primary data storage is on some 40,000 8-mm tapes while the most frequently used data is on nearly 300 Gigabytes of SCSI disks. Data is served to VMS and UNIX analysis clusters over an FDDI network from a centralized file server. We also describe plans for handling a future data set anticipated to be an order of magnitude larger. Some of the ideas being considered are alternative data structures, parallel disk access, automated tape libraries, and centralized analysis servers.