Introducing NERSC News

NERSC News is a new publication highlight-ing recent achievements by staff members in Berkeley Lab’s National Energy Research Scientific Computing Center Division. Initially, NERSC News will be distributed every other month via email and may be freely distrib-uted. NERSC News is edited by Jon Bashor, JBashor@lbl.gov or 510-486-5849.

Questions? Comments? Want to subscribe? Contact: JBashor@lbl.gov

NERSC’s QCD Library isa Model ResourceFive years ago, when the first libraries of lat-tice QCD data were established at NERSC,the idea of making such information easilyaccessible to researchers at institutionsaround the world was a bold new approach.In fact, “The NERSC Archive” format, as it iscolloquially known, has become a standardfor the lattice storage format, which madeexchanging lattices between groups muchmore efficient. The result is that lattice QCDresearchers can avoid the time consumingand computationally intensive process of cre-ating their own lattice data by drawing on theresearch of others.

Now, the concept is going global. With theappearance of Grid-based technologies, thelattice gauge theory community is develop-ing a new protocol and tools for sharing lat-tice data across the Grid. The InternationalLattice Data Federation (http://www.lqcd.org/ildg/wiki/tiki-index.php) has been formed aspart of the SciDAC initiative in the U.S. andsimilar large-scale computing projects inEurope, Japan, and Australia. Sometime thisyear the original lattice archive at NERSC willjoin similar mirrors around the world allowingaccess via Web service. Researchers will beable to query the archive database andexamine the resource catalog of all availablelattices in the world, then seamlessly trans-fer this data to their computing resourcesdistributed on the Grid.

Behind all this are hundreds of gigabytes ofnothingness, all created by lattice gauge the-orists, the particle physicists who model thebehavior of quarks. The data, called “lat-tices,” are snapshots of the way quantumchromodynamic (QCD) fields, which mediatethe forces between quarks, fluctuate in avacuum. Each lattice is a four-dimensional

News from the National Energy Research Scientific Computing Center February 2004

NEWS

With the new year came good news forNERSC users – all of NERSC’s productioncomputing and storage resources are nowGrid-enabled and can be accessed by userswho have Grid applications using the GlobusToolkit.

As a result, Grid users now have access toNERSC’s IBM supercomputer (“Seaborg”),HPSS, the PDSF cluster, the visualization serv-er “Escher,” and the math server “Newton.”Users can also get support for Grid-relatedissues from NERSC’s User Services Group.

“Now that the core functionalities have beenaddressed, our next push is to make the Grideasier to use and manage, for both end usersand the administrators who handle applica-tions that span multiple sites,” said SteveChan, who coordinated the Grid project atNERSC.

One of the major challenges faced byNERSC’s Grid team was installing the neces-sary software on Seaborg, which operatesusing a complicated software stack.Additionally, the system needed to be config-ured and tested without interfering with itsheavy scientific computing workload. Byworking with IBM and drawing upon publicdomain resources, Seaborg became accessi-ble via the Grid in January.

“With Steve Chan’s leadership and the techni-cal expertise of the NERSC staff, we crafted aGrid infrastructure that scaled to all of our sys-tems,” said Bill Kramer, general manager of theNERSC Center. “As the driving force, he fig-ured out what needed to be done, then pulledtogether the resources to get it done.”

To help prepare users, center staff presented

tutorials at both the Globus World andGlobal Grid Forum, as well as presentedtraining specifically for NERSC users.

When authenticated users log into NIM(NERSC Information Management), they arenow able to enter their certificate informationinto the account database and have thisinformation propagated out to all the Gridenabled systems they normally access,superseding the need to have separatepasswords for each system. The next step isfor NIM to help non-credentialed usersobtain a Grid credential.

Because the Grid opens up a wider range ofaccess paths to NERSC systems, the secu-rity system was enhanced. Bro, the LBNL-developed intrusion detection system whichprovides one level of security for NERSC,was adapted to monitor Grid traffic atNERSC for an additional layer of security.

The long-term benefits, according to Kramer,will be that NERSC systems are easier touse while security will be even higher.

“Of course, this is not the end of the project,”Kramer said. “NERSC will continue toenhance these Grid features in the future aswe strive to continue providing our userswith the most advanced computational sys-tems and services.”

This accomplishment, which also requiredthe building of new access servers, reflectsa two-year effort by staff members. Here is alist of participants and their contributions:

Nick Bathaser built the core production Gridservers and has been working on makingthe myproxy server very secure.

All NERSC Production Systems Now on the Grid

ORNL LBNL PNNL Fusion Grid PPDG

DOE Science Grid

ANL

(continued on page 2)

(continued on page 2)

page 2

A visualization of a QCD lattice, with red and blueblobs representing instantons and anti-instantons.

NEWS February 2004

Scott Campbell developed the Bro modules that are able to analyze the GridFTPand Gatekeeper protocols, as well as examine the certificates being used forauthentication.

Shane Canon built a Linux Kernel module that can associate Grid certificates withprocesses, and helped deploy Grid services for the STAR and ATLAS experiments.

Steve Chan led the overall project and coordinated the effort, as well as developingand porting much of the actual software that created these abilities.

Shreyas Cholia built the modifications to the HPSS FTP daemon so that it supportsGSI, making it compatible with GridFTP. He has also been working on the GlobusFile Yanker, a Web portal interface for copying files between GridFTP servers.

Steve Lau works on the policy and security aspects of Grid computing at NERSC.

Ken Okikawa has been building and supporting Globus on Escher to support theVisPortal application.

R.K. Owen, Clayton Bagwell and Mikhail Avrekh of the NIM team have worked onthe NIM interface so that certificate information gets sent out to all the cluster hosts.

David Paul is the Seaborg Grid lead, and has been building and testing Globusunder AIX.

Iwona Sakrejda is the lead for Grid support of PDSF and developed training for allNERSC users, as well as supporting all the Grid projects within the STAR andATLAS experiments.

Jay Srinivasan wrote patches to the Globus code to support password lockout andexpiration control in Globus.

Dave Turner is the lead for Grid support for Seaborg users. He recently helped auser, Frank X. Lee, move a few terabytes of files from the PittsburghSupercomputing Center to NERSC’s HPSS using GridFTP.

NERSC Helps Make Strides TowardOpen Science GridIn addition to making all of its production systems availablevia the DOE Science Grid, NERSC is also participating in theOpen Science Grid initiative. Late last year, NERSC’s 400-processor PDSF cluster was integrated into a broad scientificgrid currently connecting 27 sites in the U.S. and SouthKorea.

Driven by the high energy physics community to help pre-pare for participation in multi-institution experiments on theLarge Hadron Collider at CERN, the Open Science Grid(OSG) aims to tie together the scientific grid infrastructure ofDOE, the National Science Foundation and other agenciesin the United States.

As a lead in to the OSG, the Grid2003 project waslaunched as a collective effort by the U.S. Physics GridProjects (DOE-funded PPDG, NSF-funded iVDGL, NSF-funded GriPhyN) and the software and computing projectsof the US-ATLAS, US-CMS, the LIGO and SDSS experi-ments. Getting all 27 sites, including NERSC, integratedinto Grid2003 entailed many teleconferences, countlessemails, extensive diagnosing of cryptic error messagesand very long hours.

At NERSC, Shane Canon installed and configured grid-monitoring software, then configured the appropriate secu-rity settings for the grid middleware so that PDSF couldparticipate effectively and securely in this shared grid fab-ric. Iwona Sakrejda installed and configured the grid mid-dleware and applications packaged for Grid2003 on PDSF.

array (283 x 96, say) of four 3 x 3 complexmatrices representing these fields in a tinybox of space measuring about 2 femtometerson a side (1 fm = 10-15 m) and extendingabout 10-22 seconds in time. Lattice gaugetheorists generate many such lattices andthen use these to compute quantum aver-ages of various physical quantities underinvestigation.

To the right is a visualization of one of theselattices, showing instantons, fluctuations of thegluon field which are topologically non-trivialand strongly effect the dynamics of the quarks.Red and blue blobs represent instantons andanti-instantons, which twist with oppositetopology. In fact, it is believed that much of themass of the neutron and proton is related tothese excitations. The full four-dimensionalvisualization of these fluctuations can beviewed at http://cliodhna.cop.uop.edu/~het-rick/timeseq1135.mpg as an MPEG movie.

The most time-consuming part of latticegauge theory is the production of these lattices, which must be done so that the fluc-tuations are truly representative of the under-lying laws of quantum field theory, including,for example, the probability that quarks andanti-quarks pop out of the vacuum and then

annihilate each other, producing more gluons.Running on a computer capable of performingseveral hundred gigaflop/s, this process typi-cally takes from hours to days per lattice. Butto complete a study of a particular quantity,several hundred to thousands of lattices mustbe generated. The bottom line? Creatingenough lattices for meaningful research canrequire teraflop-years of computer time.

Once generated, however, these lattices canbe used by many researchers who might eachbe interested in studying different phenomena.Measuring quantum averages on the ensem-ble can be done quickly on local clusters oreven workstations. The obstacles to sharinglattices in the past included the size and loca-tion of the data. Typically, lattices are generat-ed at a supercomputing center, and access isrestricted to those with accounts at the cen-ters. Thus it is impossible for a researcher inFrance, say, to copy the files to their localaccount.

In 1998, Jim Hetrick at the University of thePacific in California, Greg Kilcup at The OhioState University, and NERSC’s mass storagegroup developed the first Web-based latticerepository, thereby giving other scientists accessto lattice data stored on HPSS at NERSC. The lat-

tice archive (http://qcd.nersc.gov) allowsresearchers around the world to browse and down-load lattices of interest. Hetrick and Kilcup maintainthe archive and contribute the lattices of theirresearch groups and contributions from others.

This archive was the first of its kind in the lattice gauge theory community and quicklybecame very popular. It has served tera-bytes of data to researchers around theworld who would either not have been ableto carry out their investigations, or wouldhave first had to spend many months gener-ating their own lattice data.

For more information, contact Nancy Meyerat NLMeyer@lbl.gov.

QCD Library (continued from p.1)

Production Systems (continued from p.1)