Did I Say Terabyte? I Meant Petabyte: Data Archiving in the Era of SDO
J. B. Gurman, Richard R. Fisher · AGU Spring Meeting Abstracts · 2001
Two years ago (Gurman 1999, Bull, AAS, 31, 955), we discussed the treatment of archives of the order of 10 Tbyte per year from solar physics missions in the period 2004 - 2006 (e.g. Solar-B and STEREO). By early 2007, we expect that the Sun-Earth Connections community will have to deal with data sets from the Solar Dynamics Observatory (SDO) of order I Tbyte per day. As in the previous work, we examine several alternatives for dealing with data flow and service on a fire-hose scale, and show that off-the shelf, network-attached storage can provide an inexpensive and scaleable solution. We discuss some of the differences between an SDO data archive, as well as the range of requirements for data integrity, disaster recovery, &c. in various scenarios for archive concentration or distribution.