In the old days people used index cards to sort information such as names or addresses by alphabetical order. Have you ever sorted a thousand cards? Try sorting the information on 210 DVDs or 1,422 CDs – that’s how much information is contained in a terabyte (1000 gigabytes or one million megabytes). Obviously this is something computers do better than people, much better. How much better? The new world record for sorting a terabyte of data, recently achieved by a team from the University of California San Diego, is sixty seconds – one minute flat. The same team also tied for another sorting record, that of processing one trillion data records in 172 minutes (i.e. a trillion pieces of information sorted in less than three hours).

Sorting is so important to the business of computing that it’s like a competitive sport. An independent group of computer scientists operates the Sort Benchmark Organization and several sorting contests. There are several contests because sorting like so much in computing isn’t simple. For one thing the organization makes a big distinction between sorting done in specially constructed systems, where the hardware, operating system, and application software are completely integrated, tuned, and optimized for a particular kind of sorting. This is called the Indy class of sorting. Then there is the Daytona class, which requires sorting be done only on machines and operating systems that are normally found in data centers – ‘real world’ environments.

Sorting in a computer system is always a matter of balancing components. Disk storage, memory (RAM), and processors are among the components on the hardware side. Then the operating system, which integrates the hardware capabilities, needs to work with the programming that does the sorting. In the elaborate and custom designed systems used for the Indy contests, there is usually more than one computer involved (often with multiple processors). For example, the system (dubbed TritonSort) used by the University of California San Diego team consisted of 47 nodes (computers) each with 2 Quadcore processors, 24 GB memory, and sixteen 500GB disks. Special switching and networking equipment was used to tie the system together, along with specialized software incorporated into the operating system to coordinate the sorting.

Typically the heart of a sorting test is carried out by leading edge programming – sorting algorithms – that are specially designed to shuffle data between hard disks, memory (RAM), and processors where the routines for sorting are carried out. This kind of operation is so complex that it’s a good stand-in for many kinds of computer activities. So above and beyond the importance of sorting in general computing, the ability of a system to execute fast, efficient sorts is considered one of the ways to measure overall computer capability. That’s also why the continually falling records are a good indicator of the progress of computer hardware and software.

“We’ve set our research agenda around how to make this better…and also on how to make it more general,” said UC San Diego computer science PhD student Alex Rasmussen, the lead graduate student on the team.
…
“Sorting is also an interesting proxy for a whole bunch of other data processing problems. Generally, sorting is a great way to measure how fast you can read a lot of data off a set of disks, do some basic processing on it, shuffle it around a network and write it to another set of disks,” explained Rasmussen. “Sorting puts a lot of stress on the entire input/output subsystem, from the hard drives and the networking hardware to the operating system and application software.”

[Source: Machines Like Us]