SPsort: How to Sort a Terabyte Quickly
Jim Wyllie · 1999
In December 1998, a 488 node IBM RS/6000 SP * sorted a terabyte of data (10 billion 100 byte records) in 17 minutes, 37 seconds. This is more than 2.5 times faster than the previous record for a problem of this magnitude. The SPsort program itself was custom-designed for this benchmark, but the cluster, its interconnection hardware, disk subsystem, operating system, file system, communication library, and job management software are all IBM products. The system sustained an aggregate data rate of 2.8 GB/s from more than 6 TB of disks managed by the GPFS global shared file system during the sort. Simultaneous with these transfers, 1.9 GB/s of local disk I/O and 5.6 GB/s of interprocessor communication were also sustained. Introduction The speed of sorting has long been used as a measure of computer systems I/O and communication performance. In 1985, an article in Datamation magazine proposed a sort of one million records of 100 bytes each, with random 10 bytes keys, as a useful measu...