Storage options for petabytes of data
Lisa Paton, Jonathan Cirtain, Paul Grant · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · 2008
New instrumentation that produces extremely large quantities of data presents a challenge in data processing and management. The Solar Dynamics Observatory (SDO) will house instruments that will produce 1.4TB of data per day. Processing and storing that quantity of data is a serious challenge. The instrument team for the Atmospheric Imaging Assembly (AIA) that will fly on SDO spent the last 2 weeks in September doing a large-scale side-by-side comparison of archive equipment from Apple, BlueArc, EMC, Network Appliance, SGI and Sun Microsystems. Each vendor provided 100TB of SATA disk space and the required servers to showcase their unique solutions to the problem of petabyte sized archives. The results of the testing demonstrate some of the options available in this arena. We will discuss the results of the testing, the differences and similarities between the vendors and the applicability of the technologies to various environments.