ISP Essentials Workshop -IXP
Transcript of ISP Essentials Workshop -IXP
7/11/19
1
1
ISP Essentials Workshop - IXP
Manila, Philippines8-12 July 2019
22
Content• Internet Peering
• IXP Design Considerations• How to Build an IXP
• Connecting to an IXP
• IXP Best Common Practices
7/11/19
2
3
INTERNET PEERINGModule 1
44
Customer’s Expectation
7/11/19
3
55
Or maybe this
66
IP Transit• Provide access to “The Internet”• Requires a circuit to an “upstream” ISP
– Could be local (domestic) or international– Submarine circuits are fixed capacity, not tied to usage
• Also requires service from the “upstream” ISP– Billing is based on usage, typically 95th percentile– Or based on the speed of the connection (rate-limited or not)
• Repeat to get the level of redundancy required– Two circuits to the same “upstream” ISP– Circuits to two, or more, “upstream” ISPs
7/11/19
4
77
But it’s really just…
88
Until this happens
7/11/19
5
99
Or this2 core1-vl400.vcc.kidanet.com.fj (113.20.64.49) 327.221 ms 204.824 ms 12.070 ms
3 202.170.41.85 (202.170.41.85) 1.564 ms 2.537 ms 1.383 ms
4 202.170.33.2 (202.170.33.2) 2.826 ms 2.738 ms 1.563 ms
5 gi0-2-1-4.rcr21.b001848-1.sjc01.atlas.cogentco.com (38.122.92.249) 123.644 ms 123.736 ms 123.017 ms
6 be2063.ccr21.sjc01.atlas.cogentco.com (154.54.1.161) 124.323 ms
be2095.ccr22.sjc01.atlas.cogentco.com (154.54.3.137) 124.578 ms
be2063.ccr21.sjc01.atlas.cogentco.com (154.54.1.161) 123.759 ms
7 be3144.ccr41.sjc03.atlas.cogentco.com (154.54.5.102) 124.418 ms 124.695 ms
be3142.ccr41.sjc03.atlas.cogentco.com (154.54.1.194) 123.785 ms
8 zayo.sjc03.atlas.cogentco.com (154.54.10.194) 126.692 ms 125.425 ms 124.132 ms
9 ae16.cr2.sjc2.us.zip.zayo.com (64.125.31.14) 126.694 ms 123.851 ms 124.828 ms
10 ae27.cs2.sjc2.us.eth.zayo.com (64.125.30.232) 142.824 ms 142.947 ms 142.736 ms
11 ae3.cs2.sea1.us.eth.zayo.com (64.125.29.41) 142.369 ms 142.763 ms 142.015 ms
12 ae28.mpr1.sea1.us.zip.zayo.com (64.125.29.105) 142.880 ms 144.592 ms 142.519 ms
13 64.125.193.130.i223.above.net (64.125.193.130) 162.471 ms 163.139 ms 162.358 ms
14 xe-1-0-1.pe2.brwy.nsw.aarnet.net.au (202.158.194.120) 163.443 ms 162.016 ms 163.059 ms
15 ae9.bb1.a.syd.aarnet.net.au (113.197.15.57) 162.210 ms 163.574 ms 162.243 ms
16 ge-1-1-0.bb1.a.suv.aarnet.net.au (202.158.194.226) 198.100 ms 197.932 ms
1010
What’s wrong with this picture?• Fintel customer in Suva• Accessing content at the
University of the South Pacific in Suva
• Packet travels > 25,000km• Physical distance < 10km• Adding latency• Possibly jitter too• Using expensive submarine
capacity
7/11/19
6
1111
Interconnection (aka Peering)• Connection to a “peer” network
– Exchange of traffic to customers of each peer
• Requires a circuit to the peer (or to an Internet Exchange)– Fixed cost based on capacity of the link– May also require a cross connect in a data centre– Could be fixed cost or more likely monthly recurring fee
• Traffic is settlement free mostly• Cost is the same if zero bytes exchanged or link saturated
– Don’t saturate the link, customers will be grumpy J
1212
We compete, why interconnect?• International Connections…
– If satellite, RTT is around 550ms per hop– Compared to local traffic < 10ms round trip
• International bandwidth– Costs significantly more than domestic bandwidth– Don’t congest it with local traffic
• Wastes money– Harms overall performance (end-user experience)
• Lose-lose if not interconnect locally
1
7/11/19
7
1313
Private Interconnect
1
ISP A
ISP B
Autonomous System 99
Autonomous System 334
border border
1414
Interconnection (aka Peering)• Local (loop) connections
• Not in a customer/transit relationship• Sharing customer & infrastructure routes only
– Routes that generate revenue for you
• Share costs– Two circuits, pay for one each
7/11/19
8
1515
Results of Peering• Both save money
• Local traffic stays local• Better performance, better QoS, …
• Expensive international bandwidth available for actual international traffic
• Everyone is happy (except submarine cable and satellite owners)
• It is win-win
1616
Scaling peering• What happens when new ISPs enter the equation?
– Just repeat the process?
• Private peering means that each ISP has to buy circuits to every other peer (perhaps 2 for redundancy)– For (n) peers in total, each peer needs (n-1) half circuits
• Eg 10 peers in total => 9 half circuits for each
7/11/19
9
1717
Why an Internet eXchange Point (IXP)• Private peering relies on just the two parties making best use of
the circuit– by building dedicated circuits to each peer
• n peers in total => n(n-1)/2 circuits in total
• With an IXP:– Every participant has to buy just one whole circuit
• From their premises to the IXP fabric– Improve latency performance between peers where traffic volume
wouldn’t justify a dedicated circuit– Maximizes the opportunity to fill the circuit
• Peak traffic may not be the same across all peers
1818
Internet eXchange Point (IXP)• Need a location or facility that ISPs can access and can
connect to each other over a common shared media– Eg: Ethernet switch
• Should be a NEUTRAL venue
• Needs to have multiple telco circuit providers and/or allow any licenced provider to install services
• Needs controlled environment & access
7/11/19
10
1919
Internet eXchange Points• Variety of shapes and sizes
– Commercial– Community– Tbps to Mbps– Single location or Metropolitan Area scoped– Purely a traffic exchange– Value added services
• Layer 2 exchange point– Ethernet Switches (100Gbps/10Gbps/1Gbps/100Mbps)
2020
Internet eXchange Point• Border routers in different Autonomous Systems
IXP
ISP1
ISP2
ISP3
ISP4
ISP5
ISP6
7/11/19
11
2121
Layer 2 Exchange
ISP 1 ISP 2ISP 3
IXP ManagementNetwork
ISP 6 ISP 5 ISP 4
Ethernet Switch
IXP Services:Root & TLD DNS,Routing Registry
Looking Glass, etc
2222
Layer 2 Exchange
ISP 1 ISP 2ISP 3
IXP ManagementNetwork
ISP 6 ISP 5 ISP 4
Ethernet Switches
IXP Services:Root & TLD DNS,Routing Registry
Looking Glass, etc
7/11/19
12
2323
Layer 2 Exchange• Two switches for redundancy
• ISPs use dual routers for redundancy or load-sharing• Offer services for the “common good”
– Internet portals and search engines– DNS Root & TLDs, NTP servers– Routing Registry and Looking Glass
2424
Layer 2 Exchange• Requires neutral IXP management
– Usually funded equally by IXP participants– 24x7 cover, support, value add services
• Secure and neutral location
• Configuration– Private address space if non-transit and no value add services– Otherwise public IPv4 (/24) and IPv6 (/48, /56, /64)– ISPs require ASN, basic IXP does not
• Route Servers need ASN
7/11/19
13
2525
Layer 2 Exchange• Network Security Considerations
– LAN switch needs to be securely configured– Management routers require AAA authentication, vty security– IXP services must be behind router(s) with strong filters
2626
Types of Peering• Private Peering
• Bi-lateral Peering• Multi-lateral Peering
7/11/19
14
2727
Private Peering• Dedicated circuit between two peers
– Can use a cross connect within a data centre– Or via dark fibre, telco circuit, microwave, …
• Used where traffic levels high between two peers
• Expensive, cost shared between only two parties– Often in pairs; each peer pays for one
• But ultimate in control
2828
Bi-lateral Peering• Uses an Ethernet switch at an Internet Exchange• Single cross connect to the switch
– Peer can be remote (e.g. using Metro-Ethernet)
• Dedicated BGP peering between two peers• Relies on the IXP to manage the switch• Bandwidth shared by multiple peering relationships• But direct relationship between the two peers
– More control (granularity)– If bad things happen can turn down BGP on one peer
7/11/19
15
2929
Multi-lateral Peering (MLPA)• Uses an Ethernet switch at an Internet Exchange• Single cross connect to the switch• Single BGP peering session to a “route server”• Easiest to setup, only one session
– Automatically peer with everyone else
• Reliant on IXP for both switch and route server• Relationship is with the IXP• Lesser control (granularity)
– If a peer has a problem less options to workaround
3030
Types of Peering Policy• Open
• Selective• Restrictive
7/11/19
16
3131
Open Peering• “Have a pulse peering”
• Will peer with anyone– Typically bi-lateral or multi-lateral at an existing facility– Negligible additional cost so why not?
• Typically content providers have open peering policy
3232
Selective Peering• Conditional peering
– Ex: at an IXP, will ONLY peer bilaterally and NOT with the RS
• Some negotiation may be necessary• May have some rules that peers must fulfil
– volumes, ratios, number of multiple connects
• May only peer outside of primary market
7/11/19
17
3333
Restrictive Peering• Rules!
• Has a (written) policy that defines if they will peer– Often with rules, which are set so that they don’t peer
• Often involves a minimum level of traffic– Could require a test peering to check conformance
• Also can include a “ratio” in/out traffic levels
3434
Cost tied to circuit size (not byte count)• Peering is typically settlement free
– No charge for the traffic exchanged
• Cost to peer– Router interface– Circuit to the peering fabric– Charges imposed by the IXP– All fixed, either capital expenditure or monthly recurring fee
7/11/19
18
3535
Choosing a IXP• Some markets have more than one
• Even if there is only one IXP it might appear in multiple locations– E.g. LINX is built on two rings through multiple data centres across
London
• Best location might be dictated by availability of IPLC, transit, or other factors
3636
Which IXP?• How many routes are available?
– How many other operators/providers are at the IX?– What is the traffic to and from these destinations, and how much will it reduce the
transit cost?
• What is the cost of co-lo space?– Availability of power, type of cabinet, …
• What is the cost of a circuit to the location?– If similar to transit costs are you getting a benefit?
• What is the cost of remote-hands?– For maintenance purposes to avoid serious outages
7/11/19
19
3737
Remote locations• If building to a remote location
• Make sure remote hands work at times when it’s important to you– Their 9-5 is not normally your office hours
• Check the skill set of the remote hands– Maybe engage a local consultant to help
3838
Worked Example• Single International Transit versus Local IXP + Regional
IXP + Transit
7/11/19
20
3939
Worked Example• ISP A is local access provider
– Some business customers (around 200 fixed links)– Some co-located content provision (datacentre with 100 servers)– Some consumers on broadband (5000 DSL/Cable/Wireless)
• They have a single transit provider– Connect with a 16Mbps international leased link to their transit’s PoP– Transit link is highly congested
4040
Worked Example (2)• There are two other ISPs serving the same locality
– There is no interconnection between any of the three ISPs– Local traffic (between all 3 ISPs) is traversing International
connections
• Course of action for our ISP:– Work to establish local IXP– Establish presence at overseas co-location
• First Step– Assess local versus international traffic ratio– Use NetFlow on border router connecting to transit provider
7/11/19
21
4141
Worked Example (3)• Local/Non-local traffic ratio
– Local = traffic going to other two ISPs– Non-local = traffic going elsewhere
• Example: balance is 30:70– Of 16Mbps, that means 5Mbps could stay in country and not congest
International circuit– 16Mbps transit costs $50 per Mbps per month
• local traffic charges = $250 per month, or $3K per year for local traffic
– Circuit costs $100K per year => $30K is spent on local traffic
• Total is $33K per year for local traffic
4242
Worked Example (4)• IXP cost:
– Simple 8 port 10/100 managed switch plus co-lo space over 3 years could be around US$30K total => $3K per year per ISP
– One router to handle 5Mbps would be around $9K, good for 3 years => $3K per year
– One local 10Mbps circuit from ISP location to IXP location would be around $5K per year, no traffic charges
– Per ISP total: $11K– Somewhat cheaper than $33K– Business case for local peering is straightforward - $22K saving per
annum
7/11/19
22
4343
Worked Example (5)• After IXP establishment
– 5Mbps removed from International link– Leaving 5Mbps for more International traffic – and that fills the link
within weeks of the local traffic being removed
• Next step is to assess transit charges and optimise costs– ISPs visits several major regional IXPs– Assess routes available– Compares routes available with traffic generated by those routes
from its NetFlow data– Discovers that 30% of traffic would transfer to one IXP via peering
4444
Example: South Asian ISP @ LINX• Date: May 2013
• Data:– Route Server plus bilateral peering offers 70K prefixes– IXP traffic averages 247Mbps/45Mbps– Transit traffic averages 44Mbps/4Mbps
• Analysis:– 85% of inbound traffic comes from 70K prefixes available by peering– 15% of inbound traffic comes from remaining 380K prefixes from
transit provider
7/11/19
23
4545
Example: South Asian ISP @ HKIX• Date: May 2013
• Data:– Route Server plus bilateral peering offers 67K prefixes– IXP traffic is 159Mbps/20Mbps– Transit traffic is 108Mbps/50Mbps
• Analysis:– 60% of inbound traffic comes from 67K prefixes available by peering– 40% of inbound traffic comes from remaining 383K prefixes from
transit provider
4646
Example: South Asian ISP• Summary:
– Traffic by Peering: 406Mbps/65Mbps– Traffic by Transit: 152Mbps/54Mbps
– 73% of incoming traffic is by peering– 55% of outbound traffic is by peering
7/11/19
24
4747
Example: South Asian ISP• Router at remote co-lo
– Benefits: can select peers, easy to swap transit providers– Costs: co-lo space and remote hands
• Overall advantage:– Can control what goes on the expensive connectivity “back to home”
4848
Value propositions• Peering at a local IXP
– Reduces latency & transit costs for local traffic– Improves Internet quality perception
• Participating at a Regional IXP– A means of offsetting transit costs
• Managing connection back to home network
• Improving Internet Quality perception for customers
7/11/19
25
49
IXP DESIGN CONSIDERATIONSModule 2
5050
IXP Capital Expenses• Managed Ethernet Switch(es) – mandatory
• Route Server(s) – desirable• Router(s) and server(s) to support other IXP services –
optional
7/11/19
26
5151
IXP Operating Expenses• Data Center space, including
– Rack Space– Appropriate Electricity (AC or DC, with UPS/genset)– Cooling– CCTV Camera and other physical security measures– Cross-connects
• Administration and Technical support
• Equipment maintenance• Internet transit for other IXP services – optional
5252
IXP Organization Model• Possible Models include:
– Donation / sponsorship– Cost sharing by participants
• Fixed fee, Port speed based fee, …– Membership based organization– Volunteers vs Outsourcing vs Staff
• Or any combinations of above• Neutrality is the most important, even at the beginning• Long term sustainability is the hard part so sustainable
financial model has to be established in due course
7/11/19
27
5353
Neutral Location as Starting Point• May choose one of the followings as starting point:
– University– Technology Park– Carrier Neutral Data Center – Government Data Center– Submarine Cable Station
• Having multiple carrier options is the most important• Should maintain neutrality continuously• Expansion to multiple sites can be done gradually, coupled
with growth
5454
Requirements of IXP Site • Proximity to the networks of the potential members /
participants
• Options, availability, capacity and reliability of fiber carriers
• Support for additional fiber carriers
• 24x7 access for IXP authorized support personnel
7/11/19
28
5555
Requirements of IXP Site • Availability and stability of electricity supply, including UPS
and backup power generator– Do you need DC power?
• Sufficient cooling facilities
• Good physical security – 24x7 surveillance and access control
• Availability of additional rack space for future growth
5656
General Guidelines – Governance• Organisation-wise, multi-stakeholder bottom-up approach is
the preferred approach for maximum acceptance of the community while government support is critical
• IXP should be as inclusive as possible in order to provide maximum benefits to the whole community which it serves
• Should be fair and consistent to every participant
• Should be open and transparent as much as possible
7/11/19
29
5757
General Guidelines - Geography• IXP should be within a metro area
– to avoid competing with participants and to maintain better neutrality
• Should start with the city with the largest concentration of ISPs first and gradually set up separate infrastructure in other cities if needed
5858
General Guidelines - Policies• AUPs
– Acceptable Use Policy– Minimal rules for connections
• IXP should just provide the platform to facilate participants to do peering– Peering agreements left to participants instead of mandated by IXP– Route server can be set up to facilitate easy multilateral peering as
an option – But participants should always bear in mind the end goal of “Keeping
Local Traffic Local”
7/11/19
30
5959
General Guidance - Fees• Rely on donations
• Cost recovery– Fixed membership fees– Per port fees
• One-off fee or recurring
• Commercial operations
6060
Services to Offer• Services offered should avoid competing with particpants
– e.g. web hosting at an IXP is a bad idea unless all members agree to it
• IXP operations should make performance and throughput statistics available to members– Use tools such as LibreNMS and IXP Manager to produce IX
throughput graphs for member (or public) information
7/11/19
31
6161
Services to Offer• ccTLD DNS
– the country IXP could host the country’s top level DNS– e.g. “.SE” TLD is hosted at Netnod IXes in Sweden– Offer back up of other country ccTLD DNS
• Root server– Anycast instances of root servers (I.root-servers.net, F.root-
servers.net etc are present at many IXes)
• gTLD DNS– .com & .net are provided by Verisign at many IXes
6262
Services to Offer• Route Server
– Helps provide easy multilateral peering with simple BGP configuration for participants
– Technical details to be covered later on
• Looking Glass– One way of making the Route Server routes available for global view
(e.g. www.traceroute.org)– Public or members-only access
7/11/19
32
6363
Services to Offer• Content Redistribution/Caching
– For example, Akamised update distribution service
• Network Time Protocol– Locate a stratum 1 time source (GPS receiver, atomic clock, etc) at
IXP
64
HOW TO BUILD AN IXPModule 3
7/11/19
33
6565
How to setup an IXP?• The IXP core is an Ethernet switch
– Managed switch with reasonable security features including Port Security
– Large enough buffer size with no shared buffer
• Has superseded all other types of network devices for an IXP– From the smallest 12 or 24 port 1G/10G switch– To the largest switches supporting 40G/100G interfaces– Link Aggregation support is preferable
6666
How to setup an IXP?• Each ISP participating in the IXP brings a router to the IXP
location– NOTE: ISPs may connect directly to the IXP (depending on
availability of fibre connections) instead of a dedicated router at the IXP
• Router needs:– One Ethernet port to connect to IXP switch– One WAN port to connect to the WAN media leading back to the ISP
backbone– To be able to run BGP
7/11/19
34
6767
How to setup an IXP?• IXP switch located in one dedicated equipment rack
– Also includes other IXP operational equipment
• Routers from participant ISPs located in adjacent rack(s)– May NOT be needed if remote Ethernet circuits are used by all
participants
• Using SM fibres for 1/10/40/100Gbps is recommended
6868
Peering• Each participant needs to run BGP
– They need their own AS number– Public ASN, NOT private ASN
• Each participant configures external BGP directly with the other participants in the IXP– Peering with all participants– OR– Peering with a subset of participants
• AND/OR– Joining multilateral peering via the route server
7/11/19
35
6969
Routing Advice• ISP border routers at the IXP should NOT advertise default
route or the full Internet routing table
– Carrying default or full table means that this router and the ISP network is open to abuse by other IXP members
– Recommended configuration is only to carry routes offered to IXP peers on the IXP peering router
• NOTE: Some ISPs offer transit across IX fabrics– They do so at their own risk – see above
7070
Routing (more)• ISP border routers at the IXP should not be configured to
carry the IXP LAN network within the IGP or iBGP– Use next-hop-self BGP concept
• Don’t generate ISP prefix aggregates on IXP peering router– If connection from backbone to IXP router goes down, normal BGP
failover will then be successful
7/11/19
36
7171
Address Space• Smaller IXPs may use private addresses for the IX LAN
– Public address space means IXP network could be leaked to Internet which may be undesirable
– Because most ISPs filter RFC1918 address space, this avoids the problem
– But the address space used may be in conflict with the address space used internally by some ISPs
• Bigger IXPs use public addresses for the IX LAN– Address space available from the RIRs– IXP terms of participation often forbid the IX LAN to be leaked to Internet
so special care has to be taken– Does produce documentation for traceroute– Still RECOMMENDED
7272
APNIC Policy on IXP Address Space• The End-User Assignments policy caters for IXP’s Public
Address space under IXP Address Assignment – https://www.apnic.net/get-ip/faqs/ixp-address-assignment/
• It requires that IXP have minimum 3 ISPs connected and have clear and open policy for joining
• The minimum IXP Assignment is /24 of IPv4 and /48 for IPv6
7/11/19
37
7373
Hardware• Ethernet switch needs to be managed
– Unmanaged switch means an unmanaged IXP
• Insist that IXP participants bring their own router– moves buffering problem off the IXP– Avoid spanning tree and other L2 security issues– Run port-security (MAC filtering) to protect the IX– security of the ISP connection is responsibility of the ISP, not the IXP
7474
How to set up an IXP? • The hard part with establishing an IXP is NOT the technical
part, but for relevant stakeholders to come together to build a creditable governance structure for the IXP with which everyone is happy (TRUST)
7/11/19
38
75
CONNECTING TO AN IXPModule 4
7676
IX Etiquette and Hygiene• Connect using a layer 3 device• Don’t proxy ARP• No CDP, RIP, EIGRP, OSPF, ISIS
– https://ams-ix.net/technical/specifications-descriptions/config-guide
• Don’t steal default• Don’t leak the IX prefix to the Internet• Do use consistent announcements• Do register prefixes in an Internet Routing Registry
7/11/19
39
7777
Filtering Announcements• Only send infrastructure and customer routes
– Can use community tagging to easily identify them
• Filter what you accept– Route filters (use Routing Registry data)– AS path filters– Maximum prefix count
• Minimum prefix size– Typically a /24 for IPv4 (/48 for IPv6)– May special case host routes for blackhole
7878
PeeringDB• https://www.peeringdb.com
• Identifies your AS number• Provides contacts for NOC
– Very useful when using a MLPA and need to contact peer
• Shows which facilities you use for peering– IP numbers in use at those facilities
• Brief description of who you are, how to contact you, your traffic levels, type of customers, your peering policy
7/11/19
40
7979
PeeringDB for Internet eXchange• Location and contact information
• Who is there, both on fabric and for cross connect– useful for planning when building out or searching for peers
• IXP-DB under development which will sync up with PeeringDB
8080
Tools to create router configuration• Typically use Internet Routing Registry (IRR) data
– Mostly communicate with RADB, which mirrors other registry data– APNIC members can use APNIC RR-DB
• IRRToolSet– The first public tool set, current development unclear but probably
stable
• BGPQ3– Newer tool, currently actively developed– Creates filters so you will need to script using it
7/11/19
41
8181
IRRToolSet• https://github.com/irrtoolset/irrtoolset
• Can create router configurations from policy defined in Routing Policy Specification Language (RPSL)
• Uses autnum, as-set and route objects
• rtconfig – creates configuration file
• peval – queries IRR data• Example policy in autnum objects – AS2764 & AS7575
8282
BGPQ3• https://github.com/snar/bgpq3
• Creates AS path or route filters based on IRR data• Supports a variety of formats
– IOS (both “classic” and XR), JUNOS, JSON, BIRD – Can also DIY format
7/11/19
42
8383
BGPQ3 Example (IOS-XR)
% bgpq3 -PXl prefixset-as38442 AS38442P- prefix listX- IOS-XRl- name of generated entry
no prefix-set prefixset-as38442prefix-set prefixset-as3844227.123.128.0/18,43.245.56.0/22,103.244.228.0/22,183.81.128.0/20
end-set
8484
Using communities for filters• Set a community when you import a route from a customer
or create a static (aggregate) route
• Use that community to control export to peers & transit
• Don’t allow peers or transits to set it though
• Now when you add a prefix on a router it will automatically get exported on other routers without updating their prefix lists
7/11/19
43
8585
Peering is not just technical• A personal relationship helps• Support your local NOG (Network Operator Group) • If you expand to other markets try to attend their NOG or Peering
Forums• For TL networks, this might include:
– APRICOT – Asia Pacific– Peering Asia – Asia Pacific– MyNOG – Malaysia– SGNOG – Singapore– IDNOG – Indonesia– PhNOG – Philippines
86
ROUTE COLLECTORSModule 4.1
7/11/19
44
8787
What is a Route Collector?• Usually a router or Unix system running BGP
• Gathers routing information from service provider routers at an IXP– Peers with each ISP using BGP
• Does not forward packets
• Does not announce any prefixes to ISPs
87
8888
Purpose of a Route Collector• To provide a public view of the Routing Information
available at the IXP– Useful for existing members to check functionality of BGP filters– Useful for prospective members to check value of joining the IXP– Useful for the Internet Operations community for troubleshooting
purposes• E.g. www.traceroute.org
88
7/11/19
45
89
Route Collector at an IXP
Route Collector
R1
R3
R5SWITCH
R2 R4
89
9090
Route Collector Requirements• Router or Unix system running BGP
– Minimal memory requirements – only holds IXP routes– Minimal packet forwarding requirements – doesn’t forward any packets
• Peers eBGP with every IXP member– Accepts everything; Gives nothing– Uses a private ASN– Connects to IXP VLAN
• “Back-end” connection– Second Ethernet globally routed– Connection to IXP Website for public access
90
7/11/19
46
9191
Route Collector Implementation• Most IXPs now implement some form of Route Collector
• Benefits already mentioned• Great public relations tool
• Unsophisticated requirements– Just runs BGP
91
92
ROUTE SERVERSModule 4.2
7/11/19
47
9393
What is a Route Server?• Has all the features of a Route Collector
• But also:– Announces routes to participating IXP members according to their
routing policy definitions
• Implemented using the same specification as for a Route Collector
93
9494
Features of a Route Server• Helps scale routing for large IXPs
• Simplifies Routing Processes on ISP Routers• Optional participation
– Provided as service, is NOT mandatory
• Optionally uses Policy registered in IRR
94
7/11/19
48
9595
Diagram of N-squared Peering Mesh
• For large IXPs (dozens for participants) maintaining a larger peering mesh becomes cumbersome and often too hard
95
9696
Peering Mesh with Route Servers
• ISP routers peer with the Route Servers– Only need to have two eBGP sessions rather than N
RS RS
96
7/11/19
49
97
RS based Exchange Point Routing Flow
TRAFFIC FLOWROUTING INFORMATION FLOW
RS
97
9898
Advantages of Using a Route Server• Helps scale Routing for very large IXPs
• Separation of Routing and Forwarding• Simplify Routing Configuration Management on ISPs
routers
98
7/11/19
50
9999
Disadvantages of using a Route Server• ISPs can lose direct policy control
– If RS is only peer, ISPs have no control over who their prefixes are distributed to• Some IXPs provide community based filtering option
• Completely dependent on 3rd party– Configuration, troubleshooting, etc…
99
100100
Typical usage of a Route Server• Route Servers may be provided as an OPTIONAL service
– Most common at large IXPs (>50 participants)– Examples: LINX, HKIX, AMS-IX, etc
• ISPs peer:– Directly with significant peers– With Route Server for the rest
100
7/11/19
51
101101
Things to think about...• Would using a route server benefit you?
– Avoids having to maintain a large number of eBGP peers– But can you afford to lose policy control? (An ISP not in control of
their routing policy is what?)
101
102
IXP BEST PRACTICESModule 5
7/11/19
52
103103
Concept• Some Service Providers attempt to cash in on the
reputation of IXPs
• Market Internet transit services as “Internet Exchange Point”– “We are exchanging packets with other ISPs, so we are an Internet
Exchange Point!”– So-called Layer-3 Exchanges — really Internet Transit Providers– Router used rather than a Switch– Most famous example: SingTel-IX
104104
Competition• Too many exchange points in one locale
– Competing exchanges defeats the purpose– Los Angeles and Tokyo have multiple but it’s a rarity
• Becomes expensive for ISPs to connect to all of them
7/11/19
53
105105
Rules and Restrictions• IXPs try to compete with their membership
– Offering services that ISPs would/do offer their customers
• IXPs run as a closed privileged club– E.g. Restrictive membership criteria (closed shop)
• IXPs providing access to end users rather than just Service Providers
• IXPs interfering with ISP business decisions e.g. Mandatory Multi-Lateral Peering
106106
Technical Design Issues• Interconnected IXPs
– IXP in one location believes it should connect directly to the IXP in another location
– Who pays for the interconnect?– How is traffic metered?– Competes with the ISPs who already provide transit between the two
locations (who then refuse to join IX, harming the viability of the IX)– IXP spanning multiple data centres in a city work ok (e.g. LINX)
7/11/19
54
107107
Technical Design Issues• ISPs bridge the IXP LAN back to their offices
– “We are poor, we can’t afford a router”– Financial benefits of connecting to an IXP far outweigh the cost of a
router– In reality it allows the ISP to connect any devices to the IXP LAN
• with disastrous consequences for the security, integrity and reliability of the IXP
108108
Routing Design Issues• Route Server Mandated
– Mandatory multilateral peering is generally not welcomed– ISPs have no incentive to learn BGP– Therefore have no incentive to understand peering relationships, peering
policies,– Entirely dependent on operator of RS for troubleshooting, configuration,
reliability• RS can’t be run by committee!
• Route Server is mainly to help easy peering at IXPs but should not just rely on it– Should do bilateral peering as well
7/11/19
55
109109
eXchange Point Examples• AMS-IX, DE-CIX and LINX in Europe• Equinix IX, in every Equinix Data Centre• SIX in Seattle, Washington, USA• SGIX in Singapore• MyIX in Kuala Lumpur, Malaysia• BBIX, JPIX and JPNAP in Tokyo, Japan• HKIX in Hong Kong• IX Australia in Perth, Sydney, Melbourne, Brisbane• ………• All use Ethernet Switches
110110
HKIX• Started and owned by CUHK since 1995
– Neutral to various service providers
• Gradually opened up to all networks to connect– Served licensed ISPs only initially
• 8 full-time staff– Run by part-time staff initially
• Expanded to 4 different commercial data centers in 2016-17 while still maintaining neutrality
• 290+ networks connected• Peak traffic at 1.1+Tbps • Top 3 IXP in Asia Pacific now in terms of traffic volume• See: http://www.hkix.net
7/11/19
56
111111
SGIX• Not-for-profit / Independent legal entity / Membership-based • Set up with the support of Singapore Government• With full-time staff from beginning• Went through a painful start-up process when potential
participants preferring to take wait-and-see approach– Persistency helped them get through the difficult period
• 100+ networks now• Peak traffic at 180+Gbps• Has presence in 7 commercial data centers• See: https://www.sgix.sg
112112
MyIX• Non-profit / Independent Legal Entity / Membership-based• Set up with the support of Malaysian Government• Full-time staff• Governance by elected representatives of local Malaysian ISPs• Multiple nodes in multiple cities• 40 Ordinary Members (NSP licensees; with voting right) / 30+
Associate Members (without voting right) / 20+ Members by Invitation (without voting right)
• See: http://myix.my
7/11/19
57
113113
APIX • A forum for IXPs to exchange experiences.
• 26 IXPs from 17 economies• See: http://apix.asia
114114
More info about IXPs• https://www.pch.net/resources/wiki/
– Another excellent resource of IXP locations, papers, IXP statistics, etc
• https://www.internetexchangemap.com/– Tele Geography: A collection of IXPs and interconnect points for ISPs
• https://www.peeringdb.com/– Searchable database of Exchange Points, Networks & Facilities