GÉANT Network Evolution · Based on Juniper MX Platform – All layer 2 and layer 3 services are...
Transcript of GÉANT Network Evolution · Based on Juniper MX Platform – All layer 2 and layer 3 services are...
www.geant.org 1 |
www.geant.org
Mian Usman Network Architect
GÉANT Network Evolution
www.geant.org
Runs a membership association for Europe's National Research & Education Networks (NRENs) GÉANT Association
Coordinates and participates in EC-funded projects Under Horizon 2020 the financial instrument for implementing the Innovation Union, a Europe 2020 flagship initiative aimed at securing Europe's global competitiveness
Operates a pan-European e-infrastructure GÉANT network
Manages a portfolio of services for research & education EduX
Organises and runs community events & working groups TNC, task forces & special interest groups
To support collaboration and development amongst researchers, the dissemination of information & knowledge, and provide access to a portfolio of services and infrastructure resources:
2 |
www.geant.org
Membership Association
GÉANT Association supports and represents over 40 NRENs across Europe.
Together they support over 10,000 institutions and 50 million academic users.
3 |
www.geant.org
Network The GÉANT network interconnects research, education and innovation communities worldwide, with secure, high-capacity networks.
We design, plan, build and operate the large-scale, high-performance GÉANT network that connects European NRENs to each other and the rest of the world for sharing, accessing and processing the high data volumes generated by research and education communities and for testing innovative technologies and concepts.
Interconnecting Europe’s NRENs over a 500Gb highly-resilient
pan-European backbone. NRENs serve 50 million users at 10,000 institutions across
Europe. Network services: IP, Point-to-Point Services, VPN, Testbeds,
performance monitoring
4 |
Data transfer tests in 2017 between 10G servers in GÉANT and AARNET achieved 9.73Gbps over 48h through R&E networks, whereas over commercial links
this was only 1.77Gbps.
www.geant.org
GÉANT operates an advanced, high-performance network supporting Europe’s NRENs with cost-effective, highly available, high-capacity services customised for and dedicated to research and education
Availability Targets 99.999% - 27 seconds downtime/month
99.99% - 4.5 minutes/month 99.4%* - 4 hours/month
GÉANT network services
GÉANT IP – Ultra-high performance uncontended IP connectivity at up to 100Gbit/s
GÉANT VPN – Layer 3 VPN services for NRENs and institutions supporting private networking needs
GÉANT Point-to-Point – High performance dedicated connectivity up to 100Gbit/s for the most demanding applications
GÉANT Open – Allows NRENs and approved commercial organisations to exchange connectivity in a highly efficient and flexible manner
eduroam – Seamless WiFi access around the world
Network Services
5 |
www.geant.org
GÉANT IP network
Based on Juniper MX Platform – All layer 2 and layer 3 services are provided on IP layer
GÉANT Point-to-Point – 10G and 100G lambda services are provided on optical infrastructure
Makes use of GÉANT DWDM Network for its core links and uses leased capacity where dark fibre isn’t available.
6 |
GÉANT Network Topology
www.geant.org 7 |
0
500
1000
1500
2000
2500
3000
2015 2016 2017 2018
GÉANT traffic PB/year
IP/MPLS Lambdas
• 2.4 EB of data received in 2018
• 26.4% growth from 2017 (29.9% for IP/MPLS)
• Longer term trend - 30%
0
100
200
300
400
500
600
700
800
Q12015
Q22015
Q32015
Q42015
Q12016
Q22016
Q32016
Q42016
Q12017
Q22017
Q32017
Q42017
Q12018
Q22018
Q32018
Q42018
GÉANT traffic PB – Quarterly view
• Data growth x 3 over four years period
Network and Capacity Growth
www.geant.org 8 |
Network and Capacity Growth - Traffic Distribution
LAMBDAS 39%
R&E 23%
LHCONE 15%
IAS 14%
OTHERS 9%
GÉANT traffic - by category
LAMBDAS – 10G and 100G Ethernet point to point services provisioned directly over the Infinera OTN switching layer
R&E – IP/MPLS routing for Global R&E prefixes
LHCONE – Dedicated L3VPN for LHC
IAS – GÉANT on-net internet access service
OTHERS – includes other L3VPNs, L2VPNs, MDVPN, GÉANT open and LANs
www.geant.org 9 |
• Support exponential bandwidth increase • With highly unpredictable traffic • Cost optimisation for backbone link upgrades
• Improve network visibility and control operational costs
• Provide programmability • offering the ability to use APIs to link applications to the network • Increasing agility and speed of service development
• Reduce dependence on vendor (software) • Preference for using open and standard hardware APIs
• Short Funding Cycle – Higher cost of change every 7-8 years
Network Evolution - Challenges
www.geant.org
Fiber Renewal Plan Starting Point – Current Network
• Current Fiber Network:
• Short term leases
• 14 countries connected on fiber
• UK, BE, FR, CH, DE, AT, NL, HU, HR, IT, SI, SK, CZ, ES (spur)
• Other countries on leased lines
• Typical capacity multiple 10Gs
10 |
www.geant.org
GÉANT Topology - IRU SGA
EC created a funding vehicle to procure infrastructure on long terms contracts and with 100% funding.
Go beyond the state-of-the-art by restructuring the backbone network through exploration and procurement of long-term IRUs and associated equipment to increase the footprint, stimulating the market in cross-border communications infrastructure whilst decreasing the digital divide and reducing costs
Extract from objective for the IRU SGA
www.geant.org
Regional Studies - to Agree Topology
12 |
Ireland UK
UK/Ireland Led by Guy Roberts
Belarus Latvia Estonia Lithuania
Russia
Finland
Sweden Norway Denmark
Baltic - Led by Ieva Muraškienė (LITNET)
Poland
Greece Hungary Bulgaria
Romania Serbia
South East Europe - Led by Pavel Skoda (CESNET)
Albania
Slovakia
Israel
Cyprus Turkey
Macedonia
Montenegro
Croatia
France Croatia
Central Europe region - Led by Sebastiano Buscaglione
Poland Germany
Italy
Switzerland
Austria Czech Republic
Slovenia
Malta Holland Belgium UK
Spain
Hungary
Luxembourg
+ Nordunet
Portugal
Spain
Iberian Peninsula Led by Emanuel Massano (GARR)
France
Italy
www.geant.org
Stakeholders review
Network Blueprint
IRU budget €
Financial constraints
Desirable outcome?
Community Engagement and
Workshops
Regional Studies
Technology Strategy Traffic
Projections
Market RFI
26 |
Infrastructure Sharing Equipment
Cost
GÉANT topology – a network blueprint
www.geant.org
Straw Man approved at GÉANT General Assembly June 2018 • REPLACEMENT NETWORK
proposal
• De-risked: Included links that can be built with sufficient certainty
• Based on regional studies
• Investment required: 48 M€
14 |
Result: On fiber: 24 countries: UK, IE, PT, ES, FR, BE, NL, DE, EE, LV, LT, PL, AT, CZ, SK, CH, HU, IT, SI, HR, RO, BG, GR, RS CYNet, IUCC, MARNET, MREN, RASH, RESTENA, ULAKBIM: • standard leased capacity (minimally 10GE,
might be 100GE by end of project • or additional DF/spectrum projects as part
of regional improvement / extension
Commercial DF or spectrum NREN DF or spectrum
www.geant.org
Network Infrastructure and Services Operations
15 |
€-
€1
€2
€3
€4
€5
€6
€7
1.19 4.19 7.19 10.19 1.20 4.20 7.20 10.20 1.21 4.21 7.21 10.21 1.22 4.22 7.22 10.22
An
nu
al O
&M
(M
Eur)
Evolution of connectivity O&M in positive "Flip"(*) scenario
OLD Commercial DF NEW Commercial DF OLD Line System NEW Line System
NEW DCI OLD Commercial Leased NEW Commercial Leased OLD NREN Spectrum
OLD NREN Leased NEW NREN DF NEW NREN Spectrum
(*) “Flip”: Quickly changing contracts for existing fibres to IRU
www.geant.org 16 |
• Vendor and Technology Agnostic Architecture
• Disaggregate: de-couple innovation cycles and accelerate innovation • Enable GÉANT to use best-in-class solution at different layers
• Decrease dependence on a single vendor
• Increase options to connect NRENs to different platform depending on their needs • Optimize investments to align with requirements
• Reduce space and power requirements by allowing smaller form factors
Open and disaggregated architecture allows to evolve, and to differentiate from commercial providers and develop and deliver advanced networking services
for R&E users
Network Evolution Plans - Flexibility and Control
www.geant.org 17 |
Powerful planning tools - understand impact Support current investment choices and understand future impact
• Network modelled and analysed at component level • Understand impact of growth on costs
• Where, when and how expenditure is expected
• Better understanding of interplay between technical choices and network growth at both technical and financial level allows for identifying areas for improvement
• Detailed, flexible modelling allows benchmarking and evaluating alternatives
Better/More transparent decision making
Supporting evidence for review and process improvement
www.geant.org 18 |
Vendor Proprietary Network Controller
Multi-vendor Open Network Controller/Orchestrator
• Proprietary and Closed APIs • Vendor-Specific data model
Vertically integrated single vendor solution
• Open and Standard APIs
• Open data model
Multi-vendor, disaggregated
solution
NMS NMS
Open Optical Line SYSTEM
Disaggregate: Open Optical Line System
DCIs
Network Evolution - Opening up the optical transport layer
www.geant.org 19 |
• Supporting growth on current
Infinera DTNx leads to very high costs, and is not effective • Major changes expected on Network in
next few years – DTNx components may be obsolete
• Requirement-to-Technology mapping not optimal for DTNx – 70% of capacity does not use (expensive) OTN switching capabilities
• Modelling used to understand how and where DCIs should be used to integrate the Infinera system to offset short term issues and optimize investment
0 €
1 €
2 €
3 €
4 €
5 €
6 €
7 €
8 €
2 018 2 019 2 020 2 021 2 022M
ilio
ny
Cost comparison Pure Infinera vs Infinera + DCI hybrid solution
Cumulative costs (€)
Infinera DTNx only Infinera DTNx + DCI
Powerful planning tools - understand impact Guide introduction of new technology
www.geant.org 20 |
• From a monolithic block to a open and disaggregated optical layer in GÉANT network
• Procured DCI (Infinera Groove G30) • Significantly reduces the cost of upgrading the GÉANT backbone links • Significantly reduces the space and power requirements • Protects GÉANTs investment by re-using the DCI’s with different optical line system
• Next steps: OLS
Vendor Meetings
Facebook Voyager Lab testing and Field Trial
Lab Testing field trial
DCI equipment
Technical Specification
for DCI Procurement
Launch DCI
Procurement
ConcludeDCI
Procurement
Deploy DCI
Launch OLS
Procurement
Deploy OLS
Network Evolution - Opening up the optical transport layer
www.geant.org 21 |
Plan for deployment completed during 2018 - based on modelling and analysis
2 phases:
•Phase 1 (2018-2019) – Western ring IP trunks move to DCI and upgrade to 300G
•Phase 2 (2020) – Milan - Marseille – Geneva IP trunks move to DCI
First 300G link live in GÉANT from December 2018 (Lon1-Lon2)
Phase 1 overview
London Amsterdam
Frankfurt Geneva
Paris
Planning and Optimisation – Planning optimal DCI Deployment
www.geant.org 22 |
Options: (cost) effective solution?
Network Evolution - Opening up the packet layer
www.geant.org 23 |
Network Evolution - Opening up the packet layer
• Investigated alternative solutions
• Incumbent vendor’s response to changing market: a cost-effective small device • Low cost of change, as no integration work is required, no forklift migration. • Fewer new skills to learn for the Ops and NOC teams • Both software and hardware support provided by a single vendor.
• Introduction of MX204 where appropriate • Power consumption -66%, Yearly maintenance cost -79%
• Does not preclude later introduction of other vendors!
Software and
Hardware Vendor
Meetings
Worked with JRA1 to develop
evolution options
Worked with Metaswitch and Corsa to create a
new solution
Loan kit from Dell, Arista and Agema
for testing
Testing of different white boxes with IPInfusion
and Metaswitch
New Juniper Product MX204
Testing of MX204 and Cost Benefit
Analysis
Order Placed
Replace MX480 with
MX204 in 14 Locations
www.geant.org
Planning and Optimisation – Introducing the MX204
MX 204
4x QSFP – 1x100G or 4x10G
8x SFP – 1G or 10G
400G total switching capacity Enhanced QoS PTP support Dual PSU Single routing engine
~80% OPEX reduction
~70% less power
Maintenance cost could be optimised for small PoPs in GÉANT
MX204 tested for operational deployment and procured
Design for implementation of first few sites completed
Deployment planned for Q2 - 2019
www.geant.org
Infinera OTN
Juniper MX-960 and MX-480
Juniper MX204
25
NRENs
Dark Fibre Leased Capacity
Infinera
Infinera OTN
Coriant DCI
New Interconnection between routers
Juniper MX-960
Replace MX-480s With MX-
204
Open Line System
Coriant DCI
Deploy OLS and remove OTN
Packet Core Devices
Edge Devices
Simplify Packet Layer
DCI – External Transponders
Vendor Agnostic Optical Layer
Increase Options for
NREN Connectivity
Low Capex and Reduced Opex
Low space and power
requirements
66% reduction in power and
70% reduction in maintenance
Network Evolution - Architectural Impact
www.geant.org 26 |
Network Configuration
Automation and Change Management
Orchestration Policy-based
Automation of Configuration
Intent based
Significant work carried out on Automation
Engaged with several vendors - trial with
vendors planned in 2019
Long Term Goal
Users intent and network configuration
in sync
Network Evolution – Management Software and Systems
www.geant.org 27 |
Lab built to replicate the production network 1 • MPLS
• IS-IS
• Multicast
• IPv4 & IPv6
• Netflow v9
• MDVPN
• L2 circuits
• EVPN
• NSR/GRES
• DWDM
• 1/10/100G links & LAGs
• LDP
• RSVP
• QoS
• Elephant flows
• MBGP
• Logical systems
• BFD
• 1.8m routes
• 50Gbps flows
• Simulation of 8 NRENs
• Simulation of Internet
exchange points
• Enough load to test
convergence and
protocols
2 Traffic generation and full routing table
200+ tests
• Physical flapping
• Protocol behaviours
• Route propagation
• Interoperability
• Stability
• Memory leaks
3
Outcome: No service-affecting bugs have been introduced into the network for the last four years
4 Troubleshoot exceptions – report to vendor
5
6
Repeat test until all pass
Certify code for release into network
Check for expected behaviour through a series of tests
Test suite automated using Python and Robot
framework, saving three man months per
iteration. Tests can be run nightly in the lab and
any variation in results will be highlighted. Robot
keeps track of configuration changes and when
they occurred.
Testing can also be used to
verify that new
services/features do not
interfere with existing
services on GÉANT
network.
Network Test Automation – JUNOS code certification, benchmarking and
qualification
www.geant.org 28 |
Infinera OCG Band mux
EDFA EDFA
splitter splitter DCI - Groove DCI - Groove
Infinera OCG Groove
Add span noise
Plot Rx sensitivity
Infinera OCG Band mux
50km SMF VOA 1
VOA 2
0,0000002
0,000002
-20 -15 -10 -5 0
pre
-FEC
BER
Rx power dBm
Rx sensitivity v. noise @ 8QAM Spectrum
• Scripting allows to set/read all parameters • Dozens of optical settings for new transponders • Automation allows hundreds of tests to be run overnight
Vary power/ modulation/
FEC
Infinera DTN Infinera DTN
Network Test Automation – Optical Lab Automation
www.geant.org 29 |
Deliver network evolution and
forecast long-term cost of the GÉANT
network
Plan for the replacement of the
existing GÉANT fibre network
Deliver new functionality to the
existing GÉANT platforms and
document how the GÉANT network will
evolve
Develop plans for how new network
infrastructure can be used in the GÉANT
network
Evolution Plans Architecture
Cost modelling & Long-term financial
forecasting capabilities
Technology assessments
Fibre Renewal Plan
Network Evolution Plan – Summary
www.geant.org 30 |
Thank you
www.geant.org
Any questions?
© GEANT Limited on behalf of the GN4 Phase 2 project (GN4-2). The research leading to these results has received funding from the European Union’s Horizon 2020 research and innovation programme under Grant Agreement No. 731122 (GN4-2).