Here is ESnet’s idea of a 25th birthday bash: testing the ultimate high-speed network, a 100-gigabits-per-second (Gbps) prototype intended to meet the ever-increasing data demands of large-scale, collaborative science.
This year marks a quarter century since the founding of the Department of Energy’s ESnet – Energy Sciences Network – based at DOE’s Lawrence Berkeley National Laboratory (LBNL) and sponsored by the Office of Advanced Scientific Computing Research (ASCR) in the Office of Science.
From its beginning, ESnet has enabled large-scale science collaboration between DOE supercomputers and advanced instruments and global research and education institutions. It has done so by maintaining free-flowing bandwidth and reliable connections for sharing massive data sets. A researcher in Illinois, for example, can use ESnet to operate an electron microscope at LBNL’s National Center for Electron Microscopy from his desktop. The network has an international reach, allowing easy access to massive particle-smashing data sets generated at CERN’s Large Hadron Collider (LHC) on the Franco-Swiss border.
The high-speed network is always running and is used by 25,000 scientists in the United States alone. They work on major problems in such areas as energy, genomics and astrophysics – and routinely rely on ESnet in pursuit of their discoveries.
ESnet began in 1986 when the High Energy Physics network (HEPnet) merged with the Magnetic Fusion Energy Network (MFEnet). The original was based on 56-kilobit lines. Today, data generated by some of the world’s most powerful supercomputers moves at a 10-Gbps clip, and the rate is rising.
Service and technological innovation have marked ESnet’s history. For example, a software suite called OSCARS (On-demand Secure Circuit Advanced Reservation System), developed in the mid-2000s, has provided end-to-end virtual circuits for guaranteed bandwidth and performance time for experiments, a sort of reservation system for large (into the terabit range) network flows. If a genomics researcher, for instance, needs to transfer large files between a lab and a supercomputer, OSCARS can allot time to transmit and receive the data.
Bill Johnston headed ESnet when OSCARS was developed and during implementation of what is called the hybrid packet switching network, now used in virtually every research and education high-speed network. He also helped lead creation of ESnet 4, a partnership with another well-used and far-flung network, Internet2.
‘We’ll make dark fiber available to anyone who wants to put equipment on it and test it.’
The DOE Office of Science asks representatives of its scientific programs “on a rotating, yearly basis,” Johnston says, “to characterize their future research environments. In every case, you’ll find that the future involves new generations of instruments that produce vast quantities of data.”
ESnet’s average flow – a connection either by FTP, email or just a Web connection, with a subsequent disconnection – has increased steadily, Johnston says. ESnet traffic has grown by a factor of 10 roughly every 47 months since 1990.
“Large-scale science grew up with the (ESnet) network,” Johnston says. “Science instruments follow Moore’s Law – that chip density doubles every two years – like computers do, except their refresh cycle is between 10 and 15 years rather than every 18 months. As science gets harder, instruments get larger and more expensive and they generate incredible amounts of data. Only a small number of the large instruments can be built and this means that large collaborations are involved in the experiments. Further, the data volumes have grown so large that it takes storage and computing resources from all over the world to analyze the data. There is probably no aspect of science today within our Office of Science or anywhere, really, that doesn’t depend in part or completely on very-high-speed networks.”
Adds Steve Cotter, who took over from Johnston as ESnet head in 2008: “We have seen that science productivity is directly proportional to the ease of access to data. The easier we can make it to move data, the more productive scientists can be, and that will ultimately reflect how competitive the United States is.”
The new 100 Gbps network, Cotter says, will be like “having a fire hose to put out a fire rather than a string of garden hoses. If you would say that a 10 gigabit stream would be the garden hose, you would get much more efficient transfer of water if you used a fire hose that was 10 times the size.”
The prototype is part of the DOE Office of Science Advanced Networking Initiative (ANI), which will support thousands of DOE scientists and is expected to begin operation by early next year. A portion of the financing also goes to developing a national-scale network test bed, made available to researchers and industry for experiments with new network technologies, protocols and applications to help them get up to speed on the new 100 Gbps capability.
One project to be tested on the prototype is Magellan, which is studying the viability of cloud computing for science applications. Preliminary research suggests that science applications running on supercomputers as a cloud service could improve performance 6- to 20-fold over commercial cloud computing providers.
With ANI funding, DOE and ESnet have purchased significant so-called dark fiber lines for the broader scientific community. Carriers lay fiber bundles in the ground, but the network is considered dark until equipment is installed to transmit light through it. At that point it’s considered lit fiber. This dark-fiber infrastructure, Cotter says, is like a separate network that will let researchers run experiments without disrupting flow connectivity in the lit system.
“We’ll make dark fiber available to anyone who wants to put equipment on it and test it because scientists rely on ESnet for their research and experiments, and the last thing they want is for someone to put equipment on it that will cause the network to break,” Cotter says.
The prototype network, including hardware and the dark fiber, will initially connect three DOE unclassified supercomputing centers: NERSC (National Energy Research Scientific Computing Center) at LBNL and the leadership computing facilities at Oak Ridge and Argonne national laboratories, as well as MAN LAN (the Manhattan Landing International Exchange Point).
Cotter notes that capacity has been the driving force in networking and computation for decades – at the expense of energy efficiency. “Networking equipment is consuming ever greater amounts of power. If we continue on this path it will get to the point where we need small nuclear reactors to power network equipment sites.”
To ensure that ESnet remains viable for, say, another 25 years, the prototype network will be instrumented so it’s aware how the power it consumes is generated. This will allow ESnet to save energy and seek out network pathways powered by clean, renewable resources.