HPC must ramp up efficiency
to deal with extreme-data flood
(page 2 of 4)
A 3-D visualization of plasma flow in a tokmak fusion reactor. Click image to enlarge and for more information.
To do that, researchers must build continuous processes into HPC. “We must think of the entire flow, end-to-end knowledge, while designing the system itself, maybe even at the expense of computational capability,” says Alok Choudhary, John G. Searle professor of electrical engineering and computer science at Northwestern University. “We must ask how long it takes to get an answer instead of just how long it takes to do the computation (and) think about extracting data on the fly and analyzing it” as the process runs. This is called the in situ approach.
As HPC evolves toward exascale and data-intensive fields generate exabytes of data , data management, especially moving data, creates an increasingly complicated problem. A critical issue is that the speed of disk storage has not kept pace with increasing processor speeds, while network limitations constrain the speed of input/output (I/O), or the communication between computational components or between the computer and, say, a user or outside system. Also, the power cost of moving data on the computer and especially on and off the computer is very expensive. The problem, Choudhary says, extends from “just thinking about the meaning of doing I/O down to the nuts and bolts of accomplishing that.”
Some existing tools, such as the Adaptable I/O System (ADIOS), show promise for speeding up I/O. Scott Klasky, leader of the scientific data group at Oak Ridge National Laboratory and head of the ADIOS team, says the tool already works with a range of applications, from combustion to fusion to astrophysics. Nonetheless, advances in I/O also must work with many analysis and visualization tools, Klasky says.
Software tools such as ParaView and VisIt – two key Department of Energy visualization platforms – can use the ADIOS framework to run either in situ or on stored data. This enables analysis and visualization without extensive theoretical knowledge. Such tools help enormously with data-intensive science. However, they’re not expected to scale much beyond today’s computer resources and will have to be reworked on emerging hardware.
Other approaches also could accelerate I/O. Instead of writing related data to one file, for example, some systems, such as ADIOS, can write faster to multiple subfiles. “This way,” says Arie Shoshani, senior staff scientist at Lawrence Berkeley National Laboratory, “you can write better in parallel and sometimes you get an order of magnitude improvement in I/O.”