The Value of Peering - PacNOG: The Pacific Network Operators Group
Transcript of The Value of Peering - PacNOG: The Pacific Network Operators Group
© 2009 Cisco Systems, Inc. All rights reserved.ISP Workshops 1
The Value of Peering
ISP/IXP Workshops
© 2009 Cisco Systems, Inc. All rights reserved.ISP Workshops 2
The Internet
Internet is made up of ISPs of all shapes and sizesSome have local coverage (access providers)Others can provide regional or per country coverageAnd others are global in scale
These ISPs interconnect their businessesThey don’t interconnect with every other ISP (over 32000distinct autonomous networks) – won’t scaleThey interconnect according to practical and business needs
Some ISPs provide transit to othersThey interconnect other ISP networks
© 2009 Cisco Systems, Inc. All rights reserved.ISP Workshops 3
Global ISP
Global ISP Global ISP
Global ISP
$$$$$$$$$$$$$$$
Regional ISP
IXP
Access ISP
Regional ISP Regional ISP
Regional ISP
IXP
Access ISP
Access ISP Access ISP
Access ISP
Access ISP
Categorising ISPs
© 2009 Cisco Systems, Inc. All rights reserved.ISP Workshops 4
Peering and Transit
TransitCarrying traffic across a networkUsually for a feeExample: Access provider connects to a regional provider
PeeringExchanging routing information and trafficUsually for no feeSometimes called settlement free peeringExample: Regional provider connects to another regionalprovider
© 2009 Cisco Systems, Inc. All rights reserved.ISP Workshops 5
ISP 1 ISP 2
Private Interconnect
Two ISPs connect their networks over a private linkCan be peering arrangement
No charge for trafficShare cost of the link
Can be transit arrangementOne ISP charges the other for trafficOne ISP (the customer) pays for the link
© 2009 Cisco Systems, Inc. All rights reserved.ISP Workshops 6
Public Interconnect
Several ISPs meeting in a common neutral location andinterconnect their networks
Usually is a peering arrangement between their networks
IXP
ISP 1 ISP 2
ISP 3
ISP 4ISP 5
ISP 6
© 2009 Cisco Systems, Inc. All rights reserved.ISP Workshops 7
ISP Goals
Minimise the cost of operating the business
TransitISP has to pay for circuit (international or domestic)ISP has to pay for data (usually per Mbps)Repeat for each transit providerSignificant cost of being a service provider
PeeringISP shares circuit cost with peer (private) or runs circuit topublic peering point (one off cost)No need to pay for dataReduces transit data volume, therefore reducing cost
© 2009 Cisco Systems, Inc. All rights reserved.ISP Workshops 8
Transit – How it works
Small access provider provides Internet access for acity’s population
Mixture of dial up, wireless and fixed broadbandPossibly some business customersPossibly also some Internet cafes
How do their customers get access to the rest of theInternet?
ISP buys access from one, two or more larger ISPswho already have visibility of the rest of the Internet
This is transit – they pay for the physical connection to theupstream and for the traffic volume on the link
© 2009 Cisco Systems, Inc. All rights reserved.ISP Workshops 9
Peering – How it works
If two ISPs are of equivalent sizes, they have:Equivalent network infrastructure coverageEquivalent customer sizeSimilar content volumes to be shared with the InternetPotentially similar traffic flows to each other’s networks
This makes them good peering partners
If they don’t peerThey both have to pay an upstream provider for access to eachother’s network/customers/contentUpstream benefits from this arrangement, the two ISPs bothhave to fund the transit costs
© 2009 Cisco Systems, Inc. All rights reserved.ISP Workshops 10
The IXP’s role
Private peering makes sense when there are very fewequivalent players
Connecting to one other ISP costs XConnecting to two other ISPs costs 2 times XConnecting to three other ISPs costs 3 times XEtc… (where X is half the circuit cost plus a port cost)
The more private peers, the greater the cost
IXP is a more scalable solution to this problem
© 2009 Cisco Systems, Inc. All rights reserved.ISP Workshops 11
The IXP’s role
Connecting to an IXPISP costs: one router port, one circuit, and one router to locateat the IXP
Some IXPs charge annual “maintenance fees”The maintenance fee has potential to significantly influence thecost balance for an ISP
Generally connecting to an IXP and peering therebecomes cost effective when there are at least threeother peers
The real $ amount varies from region to region, IXP to IXP
© 2009 Cisco Systems, Inc. All rights reserved.ISP Workshops 12
The IXP’s role
Who peers at an IXP?
Access ProvidersDon’t have to pay their regional provider transit for local trafficKeeps latency for local traffic low‘Unlimited’ bandwidth through the IXP (compared with costlyand limited bandwidth through transit provider)
Regional ProvidersDon’t have to pay their global provider transit for local andregional trafficKeeps latency for local and regional traffic low‘Unlimited’ bandwidth through the IXP (compared with costlyand limited bandwidth through global provider)
© 2009 Cisco Systems, Inc. All rights reserved.ISP Workshops 13
The IXP’s role
Global Providers can be located close to IXPsAttracted by the potential transit business available
Advantageous for access & regional providersThey can peer with other similar providers at the IXPAnd in the same facility pay for transit to their regional or globalprovider(Not across the IXP fabric, but a separate connection)
Transit
IXP
Access
© 2009 Cisco Systems, Inc. All rights reserved.ISP Workshops 14
Connectivity Decisions
TransitAlmost every ISP needs transit to reach rest of InternetOne provider = no redundancyTwo providers: ideal for traffic engineering as well asredundancyThree providers = better redundancy, traffic engineering getsharderMore then three = diminishing returns, rapidly escalating costsand complexity
PeeringMeans low (or zero) cost access to another networkPrivate or Public Peering (or both)
© 2009 Cisco Systems, Inc. All rights reserved.ISP Workshops 15
Transit Goals
Minimise number of transit providersBut maintain redundancy2 is ideal, 4 or more is bad
Aggregate capacity to transit providersMore aggregated capacity means better value
Lower cost per Mbps4x 45Mbps circuits to 4 different ISPs will almost always costmore than 2x 155Mbps circuits to 2 different ISPsYet bandwidth of latter (310Mbps) is greater than that of former(180Mbps) and is much easier to operate
© 2009 Cisco Systems, Inc. All rights reserved.ISP Workshops 16
Peering or Transit?
How to choose?
Or do both?
It comes down to cost of going to an IXPFree peeringPaying for transit from an ISP co-located in same facility, orperhaps close by
Or not going to an IXP and paying for the cost of transitdirectly to an upstream provider
There is no right or wrong answer, someone has to do thearithmetic
© 2009 Cisco Systems, Inc. All rights reserved.ISP Workshops 17
Private or Public Peering
Private peeringScaling issue, with costs, number of providers, andinfrastructure provisioning
Public peeringMakes sense the more potential peers there are (more isusually greater than “two”)
Which public peering point?Local Internet Exchange Point: great for local traffic and localpeersRegional Internet Exchange Point: great for meeting peersoutside the locality, might be cheaper than paying transit toreach the same consumer base
© 2009 Cisco Systems, Inc. All rights reserved.ISP Workshops 18
Local Internet Exchange Point
Defined as a public peering point serving the localInternet industry
Local means where it becomes cheaper to interconnectwith other ISPs at a common location than it is to paytransit to another ISP to reach the same consumerbase
Local can mean different things in different regions!
© 2009 Cisco Systems, Inc. All rights reserved.ISP Workshops 19
Regional Internet Exchange Point
These are also “local” Internet Exchange Points
But also attract regional ISPs and ISPs from outside thelocality
Regional ISPs peer with each otherAnd show up at several of these Regional IXPs
Local ISPs peer with ISPs from outside the localityThey don’t compete in each other’s marketsLocal ISPs don’t have to pay transit costsISPs from outside the locality don’t have to pay transit costsQuite often ISPs of disparate sizes and influences will happilypeer – to defray transit costs
© 2009 Cisco Systems, Inc. All rights reserved.ISP Workshops 20
Which IXP?
How many routes are available?What is traffic to & from these destinations, and by how muchwill it reduce cost of transit?
What is the cost of co-lo space?If prohibitive or space not available, pointless choosing this IXP
What is the cost of running a circuit to the location?If prohibitive or competitive with transit costs, pointless choosingthis IXP
What is the cost of remote hands/assistance?If no remote hands, doing maintenance is challenging andpotentially costly with a serious outage
© 2009 Cisco Systems, Inc. All rights reserved.ISP Workshops 21
Example: South Asian ISP @ LINX
Facts:Route Server plus bilateral peering offers 30k prefixesIXP traffic is 18Mbps/6MbpsTransit traffic is 24Mbps/11Mbps
Analysis:42% of inbound traffic comes from 30k prefixes available bypeering58% of inbound traffic comes from remaining 250k prefixes fromtransit provider
© 2009 Cisco Systems, Inc. All rights reserved.ISP Workshops 22
Example: South Asian ISP @ HKIX
Facts:Route Server plus bilateral peering offers 20k prefixesIXP traffic is 4Mbps/1MbpsTransit traffic is 10Mbps/3Mbps
Analysis:29% of inbound traffic comes from 20k prefixes available bypeering71% of inbound traffic comes from remaining 250k prefixes fromtransit provider
© 2009 Cisco Systems, Inc. All rights reserved.ISP Workshops 23
Example: South Asian ISP
Router at remote co-loBenefits: can select peers, easy to swap transit providersCosts: co-lo space and remote hands
Servers at remote co-loBenefits: mail filtering, content caching, etcCosts: co-lo space and remote hands
Overall advantage:Can control what goes on the expensive connectivity “back tohome”
© 2009 Cisco Systems, Inc. All rights reserved.ISP Workshops 24
Value propositions
Peering at a local IXPReduces latency & transit costs for local trafficImproves Internet quality perception
Participating at a Regional IXPA means of offsetting transit costs
Managing connection back to home network
Improving Internet Quality perception for customers
© 2009 Cisco Systems, Inc. All rights reserved.ISP Workshops 25
Summary
Benefits of peeringPrivateInternet Exchange Points
Local versus Regional IXPsLocal services local trafficRegional helps defray transit costs
© 2009 Cisco Systems, Inc. All rights reserved.ISP Workshops 26
Worked Example
Single International TransitVersus
Local IXP + Regional IXP + Transit
© 2009 Cisco Systems, Inc. All rights reserved.ISP Workshops 27
Worked Example
ISP A is local access providerSome business customers (around 200 fixed links)Some co-located content provision (datacentre with 100servers)Some consumers on broadband (5000 DSL/Cable/Wireless)Some consumers on dial (1000 on V.34 type speeds)
They have a single transit providerConnect with a 16Mbps international leased link to their transit’sPoPTransit link is highly congested
© 2009 Cisco Systems, Inc. All rights reserved.ISP Workshops 28
Worked Example (2)
There are two other ISPs serving the same localityThere is no interconnection between any of the three ISPsLocal traffic (between all 3 ISPs) is traversing Internationalconnections
Course of action for our ISP:Work to establish local IXPEstablish presence at overseas co-location
First StepAssess local versus international traffic ratioUse NetFlow on border router connecting to transit provider
© 2009 Cisco Systems, Inc. All rights reserved.ISP Workshops 29
Worked Example (3)
Local/Non-local traffic ratioLocal = traffic going to other two ISPsNon-local = traffic going elsewhere
Example: balance is 30:70Of 16Mbps, that means 5Mbps could stay in country and notcongest International circuit16Mbps transit costs $50 per Mbps per month traffic charges =$250 per month, or $3000 per year for local trafficCircuit costs $100k per year: $30k is spent on local traffic
Total is $33k per year for local traffic
© 2009 Cisco Systems, Inc. All rights reserved.ISP Workshops 30
Worked Example (4)
IXP cost:Simple 8 port 10/100 managed switch plus co-lo space over 3years could be around US$30k total; or $3k per year per ISPOne router to handle 5Mbps (e.g. 2801) would be around $3k(good for 3 years)One local 10Mbps circuit from ISP location to IXP locationwould be around $5k per year, no traffic chargesPer ISP total: $9kSomewhat cheaper than $33kBusiness case for local peering is straightforward - $24k savingper annum
© 2009 Cisco Systems, Inc. All rights reserved.ISP Workshops 31
Worked Example (5)
After IXP establishment5Mbps removed from International linkLeaving 5Mbps for more International traffic – and that fills thelink within weeks of the local traffic being removed
Next step is to assess transit charges and optimisecosts
ISPs visits several major regional IXPsAssess routes availableCompares routes available with traffic generated by thoseroutes from its Netflow dataDiscovers that 30% of traffic would transfer to one IXP viapeering
© 2009 Cisco Systems, Inc. All rights reserved.ISP Workshops 32
Worked Example (6)
Costs:Router for Regional IXP (e.g. 2801) at $3k over three yearsCo-lo space at Regional IXP venue at $3k per yearBest price for transit at the Regional IXP venue by competitivetender is $30 per Mbps per month, plus $1k port charge30% of traffic offloads to IXP, leaving 70% of 16Mbps to transitprovider = $330 per month, or $5k per annumTotal with this model is $9k per year, plus the cost of the circuit(still $100k)Compare this with paying $50 per Mbps per month to the transitprovider = $10k per annum (plus cost of the circuit)
© 2009 Cisco Systems, Inc. All rights reserved.ISP Workshops 33
Worked Example (7)
Result:ISP co-locates at Regional IXPPays reduced transit charges to transit provider (competitivetender)Pays no charges for traffic across Regional IXP
Bonuses:Rate limits on router at Regional IXP Co-lo
Can prioritise congestion dependent on customer demandsInstall servers at Regional IXP co-lo facility
Filters e-mail (spam and viruses) – relieves some capacityon linkCaches content – relieves a little more capacity on link
© 2009 Cisco Systems, Inc. All rights reserved.ISP Workshops 34
Conclusion
Within the original costs of having one internationaltransit provider:
ISP has turned up at the local IXP and offloaded local traffic forfreeISP has turned up at a major regional IXP and offloaded traffic,avoiding paying transit charges to transit providerISP has reduced remaining transit charges by competitivetender at the regional IXP co-location facility
CaveatThese number are typical of the Internet todayAs ever, your mileage may vary – but do the financialcalculations first and in the context of potential technicaladvantages too