Bandwidth challenge---Transporting sloan digital sky survey data using SECTOR
Robert Grossman, Alainna C. Wonders, Yunhong Gu, Michal Sabala, Shirley Connelly, David Hanley, Joe Mambretti, Alexander S. Szalay, Ani Thakar, Jan Vandenberg · 2006
National Center for Data Mining at UICIn our SC06 BWC entry, we will transfer SDSS (Sloan Digital Sky Survey) Data Release 5 (DR5) between the SC06 show floor in Tampa and one of the NCDM labs on the UIC campus. We will use SECTOR, our newly developed distributed data space management system, to transfer DR5 in parallel between two Linux clusters in Tampa and Chicago, respectively. SECTOR transparently manages the file locating and data moving, while it employs UDT for actual data transfer. The data transfer will be from disk to disk over a 10Gb/s shared, router link between SC06 and UIC, via StarLight. We expect to reach 5Gb/s disk-to-disk data transfer rate between the two sites.