An ESnet Project Overview - Office of Science/media/ascr/ascac/pdf/meetings/may01/...ASCAC...
Transcript of An ESnet Project Overview - Office of Science/media/ascr/ascac/pdf/meetings/may01/...ASCAC...
ASCAC Presentation
James F. LeightonESnet Project Manager
Lawrence Berkeley National Laboratory
May 3, 2001
AnESnet Project
Overview
ASCAC Presentation
ESnet: In Support of DOE Science
• Mission:• Provide, interoperable, highly capable and reliable
communications infrastructure and leading-edge network services that support DOE’s missions
• Vision:• Provide seamless and ubiquitous access, via shared
collaborative information and computational environments, to thefacilities, data, and colleagues needed to accomplish their goals.
• Role: • A component of the Office of Science infrastructure critical to the
success of its research programs.
ASCAC Presentation
ESnet: What Is It?
• A nation-wide high-performance “agency mission” network— chartered to provide advanced network services to
support scientific research in DOE• Centrally funded by DOE/Office of Science
— operated and managed by project staff at LBNL• An immense (and very successful) cooperative effort• Was rated both “outstanding” and “extremely cost effective”
at the last formal Program Review• Has an extensive structure of domestic (commercial and
R&E) and international interconnects• Has a growing Advanced Technology and Research
program
ASCAC Presentation
ESnet: What Is The User Base?
• 10,000-100,000 researchers in U.S. use ESnet (guestimate)• Mostly Office of Science programs: HEP, NP, FES, BES, BER, MICS• Traffic also carried for DP and others• Involves essentially all U.S. national labs• Hundreds of universities• Hundreds of foreign institutions• Large and small collaborations (from a few to almost 2000 members)
CHARACTERISTICS OF USER BASE• Many (unaware) casual users• Variety and breadth in science• Size and geographic distribution of research community• Many data intensive and computationally intensive tasks supported
ASCAC Presentation
ESnet:An Enabler for DOE Science
• We've seen a dramatic change in the way science is done• Increase in scientific productivity - much shorter turn around time for
disseminating, assimilating and testing new ideas• Innumerable meetings via remote conference technologies• This DOE community has been a leader in use of the network for science• Network planning and deployment based on program requirements and
technological opportunities - push/ pull
ASCAC Presentation
ESnet:Network Usage
• Basic services - email, file transfer, remote login, distributed file systems• Teleconferencing - integral part of work flow, planning, coordination• Remote access to unique facilities
– experiments– supercomputers– databases– installed codes
• Collaboratories– Traditional approach - visit and/or relocate– New approach - virtual laboratories, remote participation
• Distributed computing - Grid Computing
ASCAC Presentation
ESnet:Research Life Cycle Support
• Planning and coordination• Design of experiments by distributed teams• Engineering/analysis• Online documentation• Remote Participation• Distributed code development• Distributed data analysis and
visualization
ASCAC Presentation
ESnet: A Very Brief History
• One line of ancestry can be traced back to ‘74 dial-up access to CTRCC• MFEnet, HEPnet (I.e. DECnet) & ARPAnet development continued over
’75-’85 timeframe• Memo signed in Oct, 1986 to create multi-program “ESnet”• 1987-1990: The “sociology of DOE-wide networking” barrier broken• Dec ’89: ESnet DECnet turned on• Feb ’90: Log shows security problem with hacker• Mar ’90: All T1 trunks and routers for “ESnet (1)” on-line• Feb ’92: ESnet (2) RFP released
— Jun ’94: 3 T3s turned on (LLNL-LANL, LANL-FNAL, FNAL-PPPL)— Aug ’94: Sprint master contract signed for fast-packet service
• 1995-1998: T3, OC3, and OC12 ATM connections brought on-line• Nov ’95: Decision to move ESnet and NERSC to LBNL announced• Jun ’99: ESnet (3) RFP released
— Dec ’99 Contract with Qwest signed— And a major transition is underway
ASCAC Presentation
ESnet: It’s the Sociology that’s Hard
ProgramOffices
DOE
OSC
PROGRAMS
OASCR
MICS
ESSC
ESCC
WG&TFWG&TF
WG&TFWG&TF
ESCRC
ESnetProject
• The ESnet project enjoys an excellent working relationship with both it’s technical and program “user”communities
The ESnet Steering Committee (ESSC) deals with requirements and priorities as established by DOE representative Program Principal Investigators.The ESnet Coordinating Committee (ESCC) deals with associated site and technical issues.
ASCAC Presentation
ESnet: The FY01 Budget
• The ESnet project is centrally funded by the MICS office in DOE/SC
$6.42M Base budget6.92 Communications1.19 International.35 DCS
1.00 Testbed&Research.89 Equipment
-----$16.77M FY01 MICS Funding
Additional funds are collected for special project/programsupport (mostly pass-through) now at roughly $2.5M/yr
ASCAC Presentation
ESnet: Staffing
Career ContractEngineering Services Group: 6.0Information & Services Group: 4.0 Technical Services Group: 10.0 1.0Operations : 2.0Admin Support: 1.0Management : 2.0
--- ---23.0 3.0
ASCAC Presentation
ESnet: StaffingESnet Monthly TrafficCompared to Budget
TER
AB
YTES
/Mon
th
0.00
5.00
10.00
15.00
20.00
25.00
30.00
35.00
40.00
45.00
Jan.
90
May
. 90
Sep.
90
Jan.
91
May
.91
Sep.
91
Jan.
92
May
. 92
Sep.
92
Jan.
93
May
.93
Sep.
93
Jan.
94
May
. 94
Sept
. 94
Jan.
95
May
. 95
Sep.
95
Jan,
96
May
, 96
Sep,
96
Jan,
97
May
, 97
Sep,
97
Jan,
98
May
, 98
Sep,
98
Jan,
99
May
, 99
Sep,
99
Jan,
00
May
, 00
Sep,
00
Jan,
01
E S net B udget
0
5000
10000
15000
20000
25000
FY90 FY91 FY92 FY93 FY94 FY95 FY96 FY97 FY98 FY99 FY00 FY01(P lan)
FY02(P lan)
Fiscal Y ear
$ in
$10
00E S net B udget
0
5000
10000
15000
20000
25000
FY90 FY91 FY92 FY93 FY94 FY95 FY96 FY97 FY98 FY99 FY00 FY01(P lan)
FY02(P lan)
Fiscal Y ear
$ in
$10
00
ASCAC Presentation
ESnet: Qwest Contract Overview
• Procurement for “Umbrella Services Contract”—Includes three major components
• advanced services and technology for production network• high-performance test-bed• research collaboration
—Multi-year contract (3+2+2 years), $50M+—Will overlap with existing Sprint contract
• nearly 2 year overlap possible, but not expected• includes very competitive ATM pricing
—Contract signature Dec ‘99• Transition planning & implementation underway• First sites went on-line in August, 2000• Approximately 90% done with initial transition
ESnet (Qwest) BACKBONE Apr 2001
ESnet (ESnet (QwestQwest) BACKBONE ) BACKBONE Apr 2001Apr 2001
NTON OC12OC48 ATMOC12 ATMOC3 ATMT3 ATMT1-T3 ATMT3T1-T3T1<T1Sprint Connected
6 Apr, 2001
JAPANJAPAN
TWC
TWC
TWC
JGIJGIJGI
SNLLSNLLSNLL
LBNLLBNL
SLACSLAC
Fix/MaeWest
Fix/MaeWest
Fix/MaeWest
Fix/MaeWest
YUCCAMT
YUCCAMT
YUCCAMT
BECHTEL
BECHTEL
BECHTEL
LLNLLLNL
SNV HUBSNV HUBSNV HUB
SAICSAICSAICGAGAGA
PNNLPNNLPNNLLIGOLIGOLIGOLIGO
INEELINEELINEELINEEL
LANLLANLLANL
SNLASNLASNLAALBHUBALBHUBALBHUBALBHUB Allied
SignalAlliedSignalAlliedSignalAlliedSignal
PANTEX
PANTEX
PANTEX
PANTEX
ARMARMARM
AlliedSignalAlliedSignalAlliedSignalAlliedSignal
NOAANOAANOAANOAAOSTIOSTIOSTIOSTI
ORAUORAUORAUORAUOROOROOROORO
SRSSRSSRSSRS
ORNLORNLORNL
JLABJLABJLAB
MAE-EastMAE-EastMAE-EastMAE-East
PPPLPPPLPPPLGTNGTNGTN
ANL-DCINEEL-DCANL-DCINEEL-DCANL-DCINEEL-DC
MITMITMIT
ANLANLANL
BNLBNLBNL
FNALFNALFNAL
AMESAMESAMES
CHI NAPCHI NAPCHI NAPCHI NAP
CHI HUBCHI HUBCHI HUBCHI HUB
NY-NAPNY-NAPNY-NAP
NevisColumbia
Yale
NevisColumbia
Yale
NevisColumbia
Yale
PAIX-WPAIX-WPAIX-W
RUSSIARUSSIA
QWESTQWESTATMATM
QWESTQWESTATMATM
ATLATLATLATL
4xLAB-DC4xLAB-DC4xLAB-DC
Brandeis
Brandeis
Brandeis
ITALYITALY
GERMANYGERMANY
UKUK
DANTEDANTE
NY H
UB
NY H
UB
NY H
UB
CHINACHINA
RUSSIARUSSIA
NLNLISRAELISRAEL
SINGAPORE
TAIWANTAIWANCANADACANADA NORDUNETNORDUNET
FRANCECERN
SEA HUBSEA HUBSEA HUB
SNVSNVSNV
SNVSNVSNV
SNV
SNV
SNV
SNVSNVSNV
ORN HUBORN HUBORN HUB
ORNORNORN
ORNORNORN
ORN
ORN
ORN
ALB
ALB
DC HUBDC HUBDC HUBNERSCNERSC
NY
NY
NY
KEKKEK
ORNORNORN
TECHnet
ESnetESnetDirect IP NeighborsDirect IP Neighbors
EARLY 2001EARLY 2001
JAPAN
VerioVerio
RussiaRussia
AbileneCalRENUCBUCnet
AbileneCalRENUCBUCnet
ITALYITALY
CHI-NAP60 PeersCHI-NAP60 Peers
MAE-East51+8 PeersMAE-East51+8 Peers
GermanyGermany
GWUNetwork VAvBNS
GWUNetwork VAvBNS
CHINACHINA KEKKEKRUSSIARUSSIA
NLNLISRAELISRAEL
SINGAPORE
TAIWANTAIWAN
NY-NAP38 PeersNY-NAP38 Peers
CANADACANADA NORDUNETNORDUNETFRANCE
AbileneSOXAbileneSOX
CalRENSAICSDSCvBNS
CalRENSAICSDSCvBNS
FIX/MAE-West40+6 PeersFIX/MAE-West40+6 Peers
MIX-WestAbileneDRENEI-AXExodusIBM GlobalICMLevel3Lightening InetNRENQwestSprintLinkTeleglobeVerio
MIX-WestAbileneDRENEI-AXExodusIBM GlobalICMLevel3Lightening InetNRENQwestSprintLinkTeleglobeVerio
LBL
LBL
GACGAC
LANLLANL
FIX/MAE-WFIX/MAE-W
CHI-NAPCHI-NAPCHI-NAPSTAR-TAPSTAR-TAPSTAR-TAP
ATLATLATL
MAE-EMAE-EDC-HubDC-Hub
PPPLPPPLNY-NAPNY-NAP
1 Feb, 2001
UKUK
DanteDante
NY-HUBNY-HUB
AbileneAbilene
SNV
SNV
CHICHI
CERN
ASCAC Presentation
ESnet: Collaboration Services
• Began in ’90-91 with HEP use of “excess” bandwidth on T1 trunks
• Now grown to the point that a “typical” month includes over 1200 conferences at ~2 Hours average, supporting roughly 100 conference-days of collaboration per month!
• DCS Resources now include:• Accord ISDN MCU (48 ports)• Latitude Audio Bridge (76 ports)• A web based reservation system (DCS)
• A commercial replacement for the DCS scheduler is being evaluated
• H323 (IP Video) support and interoperation with H320 (ISDN Video) now being researched and tested for system integration
All Conferences - March, 2001Number of Conferences 1279Conference Hours 2440Audio Bridge Port Hrs 5137MCU Port Hours 4155Room Hours 5529
ASCAC Presentation
ESnet: Security Considerations
• ESnet supports unclassified activity only• ESnet security responsibility falls under the auspices of the LBNL CPPM• ESnet has it’s own Project Security Officer
— point-of-contact for ESnet Project to LBNL CPPM• ESnet has a defined AUP• ESnet’s security responsibilities:
— cover ESnet resources — end at the site demarc!
• ESnet’s security requirements cover:— network utilization— component access— user services— office environment
THE ESNET
UNCLASSIFIED-SECURITY PLAN
1 Feb, 2001
ASCAC Presentation
ESnet: Research & Adv. Technology• Multiprotocol Routers
— IPv4, DECnet, OSI, X.25 at one point
• ATM— Industry catalyst for carrier deployment— First production nation-wide WAN deployment
• Advanced Protocols-one of earliest adopters of:— CIDR— BGP-4— MBGP— IPv6:
• QoS— Queue management research— Policing (CAR)— VoIP (application)
• Testbed (planned)
• PKI/Directory (proposed)
ASCAC Presentation
ESnet: Future Initiatives as Drivers
• Nano-technology• Computational materials science
network• LHC• RHIC detectors come fully
on line• Burning plasma experiment• SciDAC• SNS• Post-genome• Network research initiatives
ASCAC Presentation
ESnet: Critical Issues for the Future
• Keep ahead of the curve on domestic connectivity• Closely monitor quality of connections to universities• Continue to improve International links• Deployment and
support ofadvanced services
ASCAC Presentation
Summary
• The ESnet project thrives in an environment that— is experiencing phenomenal growth in usage— has an extremely rapid pace and broad spectrum of technology— Has simultaneous user demand for performance & reliability— is extraordinarily dependent upon wide-scale collaboration and
interaction with “peers”
• The program is highly service oriented and enjoys the trust, confidence, and support of both its technical and end user community
• ESnet is a highly successful, effective, and cooperative effort in meeting the networking and data communications requirements of the agency science research community.
ASCAC Presentation
ESnet: Internal Measurements
Include monthly, by site, mrtg
We maintain an extensive set of performance and traffic measurements
ASCAC Presentation
ESnet: Outside Measurements
Loss seen from Esnet to groups of Sites
ITU limit for lossITU limit for loss
ASCAC Presentation
ESnet: Other Services
• NTP• PGP Key Server
— over 1 million keys now— roughly 10K updates/day
• Multi-cast Backbone – now native mode on backbone• Secondary DNS• Certificate Authority, Certificate Server• Directory Services• Meeting and conference networking support
What Makes It “Fun”? (1/2)
• Moving 35+ sites without a break in service
• Economics1) Service overlap, e.g. overlap between hubs2) Termination charges3) Bridging between vendors
• Trying to forecast install dates for planning
• Every site seems to have its unique set of issues—LBL – city of Berkeley wants a franchise—ANL – required an environmental impact study—NY – power shortage in collocation bldg—NERSC – new site, operational date before circuit can be
installed
What Makes It “Fun”? (2/2)
• Hardware differences— Juniper vrs Cisco routers— Cisco vrs FORE ATM switches
• Topology differences— Hub routers vrs full BGP mesh
• Interoperability (ESnet <> ESnet3)— Tested CLIP, OSPF, BGP, MBGP, MSDP, PIM-SM
• Non-persistent bridge sites— Sites bridging between ESnet and ESnet3 will change
during the transition• But – we’re nearly there!
SNV
ALB
ORN
NYCCHI
LANL
SNLA
BNL(TELEHOUSE)
OC48-ATMOC12-ATMOC3-ATM
OC48-SONET
OC3-SONET
T3-ATM
T3
MIT
CHI-NAPPPPL
ORNLATL
SRS
ANLFNAL
AMES
PANTEX
JLAB
GTN
ASIG
T1
DC
(NY-NAP)
SNLL
LLNL
LBNL
NERSC
(FIX-W)
(PB-NAP)
(MAE-W)
OC12-SONET
SLAC
(MAE-E/ATM)
(GA)
JGI
(PNNL)
GA
(SDSC)
YUCCA-MT
(BECHTEL)
ESNET3 INITIAL CONFIGURATIONTop Level View – Qwest Access
SEA
(SAIC)
INEEL
(site) non-Qwest
ANL DCINEEL DC
4X-DC
KEK
QWESTATM
ESnet
ATM
*
***
****
* * ****
*
*
*
**
*
*
*
#
*
*
#
*
***
Installed-finalInstalled-temp#
**
20 Apr, 2001
* LLNL DC**
**
**
**
PAIX-W
*
#
#
*
**
PAIX-W* *
*
*
#
#
#
#
###
#
(NREL) *
*
#
ASCAC Presentation
ESnet: Staffing Analysis
Career ContractEngineering Services Group: 6.0Network engineering, 3rd level problem resolution (24x7), measurements & analysis, new protocols & technology system integration, documentationInformation & Services Group: 4.0Security, DCS development, testing, operation (8x5), grid servicesTechnical Services Group: 10.0 1.02nd line problem resolution, installation coordination, security, desktop & server operation & system administration, Operations: 2.0Monitoring & 1st line problem resolution (24x7)Admin Support: 1.0Management: 2.0
--- ---23.0 3.0