Why care? Peering and Transit are li3le understood and yet this is the heart of Internet infrastructure economics and it is the growth of Internet infrastructure that ul:mately funds most of us here at OFC.

I going to describe how thousands of independent organiza:ons compete but also exchange trac and con:nually grow an ever more distributed Internet infrastructure.

1

There are more than 3 billion Internet users and, with the advent of very low cost Android smart phones ($24 in India in Feb 2014), its likely that number will double in just a few years. There are also tens of millions of local networks (perhaps more than 100 million based just on the number of WiFi routers that have been sold). They connect to the Internet through tens of thousands of ISPs which may be classied as Access, Aggrega:on or Backbone, or as Local, Regional, Na:onal or Interna:onal but, of course, many ISPs cross these boundaries.

For today, Im going to focus on the 6000 or so major ISPs that form todays Internet backbone. But to understand todays complex environment, its useful to see how it emerged.

2

25 years ago, there was only one backbone. It was run by the Na:onal Science Founda:on for the benet of various researchers and government agencies. Regional networks connected to the NSF backbone but, with only one backbone, there was only one source of addressing and of ul:mate rou:ng decisions.

3

As other, commercial networks grew up, they interconnected with the NSFNET to exchange email and data les. They also found other ways to exchange data among themselves, but s:ll relied on the NSFnet as the ul:mate authority on addressing and rou:ng.

4

With the advent of the World Wide Web and the Mosaic browser, Internet growth accelerated and the NSF sought a way to get out of the backbone business. Part of this required development of a new rou:ng protocol (BGP, on which more later) which went on within the IETF between 1991-1994. Part of this required establishing four Network Access Points (NAPs) where backbone providers would exchange trac des:ned for other backbones.

Of course, each backbone provider had its own network that enabled all connected users and content providers to communicate with one another. However, users were not interested in communica:ng just with just those other users connected to the same backbone provider. They wanted to communicate with any user and any content provider, regardless of backbone provider. To oer universal connec:vity, backbone providers interconnected at NAPs (and elsewhere) to exchange trac des:ned for each others users. It is these interconnec:ons that make the Internet the network of networks that it is today.

Finally, in April 1995, the NSF stopped providing backbone services and the commercial Internet was born.

5

In order to provide complete Internet access, the backbone providers had to exchange trac with each other. Whats more, the NSFNET backbone had facilitated open trac exchange at many levels, so there were many peering agreements at rst.

But the Internet was also growing rapidly, requiring signicant capital investments. At a minimum, investors wanted to see a path to a return on their investment.

6

And, with just six-seven full backbones networks in existence, the backbone ISPs began to realize they had the makings of a cartel.

7

As a cartel, none of the backbone operators had to provide free peering to regional, local or other smaller networks. Instead they could sell them Internet Transit service a service that delivers packets to the rest of the Internet.

Gradually (and some:mes abruptly), peering rules became quite exclusive. To peer with the backbone, you had to be present at all major NAPs, you had to have a signicant amount of trac and that trac had to be roughly symmetric. This had an immediate impact on many Tier 2 operators, some of which were growing more rapidly than the backbones.

8

De-peering also impacted cable companies, several major content hos:ng networks and some large savvy content providers. These folks realized that, even if they had to buy Internet Transit from a backbone provider, they could reduce what they paid the backbone providers by exchanging trac among themselves.

9

By 2002, donut peering had emerged. The Tier 2 ISPs, cable companies and content providers had built a ring around the cartel, largely rendering the original cartel irrelevant.

10

Indeed, many Tier 2 providers now had interna:onal networks and oered lower latency &/or be3er pricing. By the early 2000s, the Internet was substan:ally more distributed.

11

The third wave, which started in the early 2000s and is s:ll evolving today, was the advent of Content Distribu:on Networks (CDNs). CDNs may have limited, private or no communica:ons infrastructure of their own, instead they distribute content servers in what is eec:vely an overlay network. Akamai and Limelight created early CDNs. Today, Google, Amazon and Level 3 also run content distribu:on networks and Nejlix has begun deploying their own CDN.

12

Typically, major CDNs supply their servers and remotely manage them, but local ISPs install them and pay for electricity and rack space. This is good business for the local ISP as it reduces latency for their customers and reduces the amount of upstream Internet transit service they must pay for.

13

The past 20 years have seen enormous turbulence among those providing the core of the Internet. The original backbone networks have survived, but their ownership has gone through a series of bankruptcies, mergers and acquisi:ons. Meanwhile the number of networks par:cipa:ng in the Internet backbone has grown from 6 to over 6000.

14

Ive been bandying around the terms Peering and Internet Transit. Let me explain exactly how they dier.

Internet Transit is a service where the upstream ISP commits to deliver trac to any valid Internet address. Its typically priced in $/Mbps/Month and the Mbps of trac is determined by measuring trac levels every ve minutes and then compu:ng the 95th percen:le of all those measurements during the month.

Now suppose Im ISP1. I have a router in a regional data center where I buy Internet Transit services, but I no:ce that 4% of my trac is to my compe:tor, ISP2, and he happens to have a router in the same regional data center just a few hundred feet away from mine. Hes my compe:tor, but we could each save 4% of our monthly bills for Internet transit if we agree to locally exchange the trac thats des:ned for each others networks.

15

No:ce that were only exchanging trac that originates with a customer of one ISP and terminates with a customer of the other peered ISP.

16

ISP2 may have other connec:ons to other ISPs, but these are not involved (or even visible) to the peering arrangement with ISP1.

Thats the key dierence. Peering is trac exchange involving only those addresses that are served by the two peers. Transit involves handling packets that will be passed o to one or more addi:onal networks.

17

But whether its peering or transit, what is actually exchanged and how does it work?

Here things are remarkably stable. Operators may exchange other kinds of trac (MPLS, Carrier Ethernet) for other services, but for Internet trac, they exchange IP packets (mostly IPv4) and they nego:ate routes using Border Gateway Protocol (BGP). IPv4 is essen:ally unchanged for over 30 years and the current version of BGP has had only minor tweaks since it was deployed 20 years ago.

Business arrangements have been turbulent, but the technology has been remarkably stable.

18

To get a be3er understanding of BGP, suppose Im running BGP on my edge router there on the lel. There are two ISPs I wish to exchange trac with (either peering or transit). In par:cular, Im interested in gemng trac to address blocks A, B & C.

My router starts by establishing BGP sessions with the edge routers at each ISP.

19

Once the sessions are up, I get an announcement from the edge router at ISP1 saying its prepared to deliver trac to address block A over a route that has three hops and trac for address block B over a route that has one hop.

20

This is followed by an announcement from ISP2 saying they can deliver trac to address block B in two hops or to address block C in two hops.

Now, I have to make some decisions.

21

First these announcements come from other organiza:ons who may or may not be competent. Should I believe ISP1 when he says he can deliver trac to address block B in just one hop?

A classic example of mistakes that can happen occurred in Feb 2008 when the government of Pakistan told Pakistan Telecom to block trac to YouTube because YouTube was hos:ng blasphemous videos. The engineers at Pakistan Telecom complied by crea:ng a very specic route for just the YouTube addresses (part of a larger Google address block). Request packets that matched this specic route were sent to a black hole server, i.e. a server that dropped each packet it received.

Unfortunately, this black hole route leaked out to the large interna:onal carrier, Hong Kong-based PCCW. PCCW didnt have route ltering in place on this par:cular link and they passed the black hole route around the world. Over 90 major ISPs erroneously accepted this route and for more than two hours YouTube was dark while almost all the worlds YouTube requests went to the black hole server in Pakistan.

So you cant always trust your neighbor, however competent they may have seemed in the past.

There are many addi:onal considera:ons. For example, certain routes may have preferen:al pricing up to a certain commitment level but become expensive at higher trac levels. So the choice of which adver:sed route to use can involve some quite complex considera:ons.

22

To give you a sense of the business trade os that go on, I have two examples.

The rst is a friend of mine who formed a xed wireless ISP in southeastern Illinois a few years ago. Because he was located in farm country, the only way he could get an Internet connec:on was by buying Internet Transit service (called Direct Internet Access or DIA) from Ameritech (now AT&T) the local telephone monopoly. His price was more than 100x what Internet Transit would have cost him in Chicago, but there were no compe:ng ber routes through his area and even if hed been close to a long distance ber route (say between Chicago and St Louis), local connec:ons to long distance ber are extremely expensive or, more olen, just not available.

Once his business was up and running, my friend spent many days driving to and from Chicago looking for tall buildings and talking to building owners. Eventually he build a series of four wireless links (totaling more than 70 miles) which connected him to Chicago. In Chicago, he signed up for a monthly recurring charge for rack space, for roof rights on the Chicago data center and for a cable from his rack to their meet me room. Hed also promised free high speed Internet service to three building owners, downstate, who gave him roof access on the route to Chicago. But now that he was connected in Chicago, he could purchase Internet transit from any of a dozen compe:ng carriers (at a :ny frac:on of what he was paying AT&T). Although he had spent nearly $100K (and untold man hours) pumng this wireless route together, he gured his payback was 9 weeks. Loca:on ma3ers!

The second thing that happened was, as his total trac grew he began to qualify for peering with major content providers like Google and Akamai. This cut further cut his costs for Internet transit.

23

The second example is only approximate, but representa:ve. I dont have the actual numbers on YouTubes trac or their costs during the 20 months between their founding in Feb 2005 and their purchase by Google in Oct-Nov 2006, but I can tell you that one of their early employees was a peering coordinator who showed up at NANOG mee:ngs early in 2006. In early 2006, there was already a great interest in peering with YouTube.

By the summer of 2006, YouTube was the 5th most tracd website in the world. They were s:ll only peering in Palo Alto, but anyone with a router in Palo Alto was interested in ooading their YouTube trac. And any Tier one carrier that didnt peer with YouTube would quickly nd trac ra:os going unbalanced on links where they handed o YouTube trac to someone who was peering with YouTube.

Im not showing YouTubes costs going to zero, but they clearly did not increase (and likely went down) as YouTubes trac grew!

24

Bill Woodcock and Vijay Adhikari of Packet Clearing House did a very comprehensive survey of backbone ISPs in 2011 gemng a remarkable 86% response rate. All the internal indica:ons are this survey yielded very high quality data.

Several interes:ng things emerged from this data. Most notably, many operators publish a set of peering requirements, and these typically include an NDA. But if you meet the requirements, there are no formal contracts! These are handshake agreements.

25

One interes:ng thing was, to the extent there are contracts between operators in dierent countries, for example the NDAs, the choice of governing law always favors the country with stable ins:tu:ons, minimum corrup:on and a func:oning judiciary.

26

In terms of how the Internet backbone is evolving, the most interes:ng thing to emerge was the rise of mul:-lateral peering. These are arrangements that started in Asia and selected loca:ons in Europe. We havent seen this in the US yet, but there is an organiza:on, open-ix.org, backed by Google and Amazon among others, that is trying to foster the spread of mul:-lateral peering.

27

Mul:-lateral peering drama:cally reduces the number of BGP sessions one must congure and manage, thus facilita:ng more peering. With bi-lateral peering, there is a separate BGP session for every peer.

28

In mul:-lateral peering, one organiza:on perhaps a co-op or a vendor provides a single route server. Each par:cipant establishes a single BGP session to this server. Typically, the route server includes session-specic congura:on which allows you some of the exibility you would have had with N bi-lateral peering sessions but, to get started, you can ignore all that and just establish one simple BGP session that reaches hundreds of peers.

29

This graph shows the number of IP addresses handled by various carriers as a func:on of how many peering agreements those carriers have. You can see that some of the original Tier 1s are s:ll visible in the upper lel, but otherwise, the Internet backbone is very distributed. And this graph is based on addresses handled, not on trac carried.

30

When we look at trac, the top ISPs are quite distributed. Also, we can see what happens when the large carrier (Level 3 at the top) buys the second largest carrier (Global Crossing in grey just below) as happened in April 2012. Both networks immediately saw a drop in trac as customers who wanted redundant connec:ons dropped one of their connec:ons to the now merged business. Then over :me, both networks see further drops in trac as the rest of the players rearrange their networks. Also, note that this trac diagram only deals with ISPs that oer Internet transit services. The second largest network in the world, by trac, is Google. If Google were shown on this graph, it would appear between the black and grey lines. So the Internet is very distributed and, as Rensys notes in their report, the rela:ve market share of the backbone carriers as a group has been falling over the past decade.

In ~20 years of the commercial Internet, no one has been able to gain control of the Internet backbone. In the 1990s, the original gang of ~six backbone providers thought they had an oligopoly (a cartel), but by 2002, second :er backbones used "donut peering to eliminate the original :er ones leverage. Since 2000, we've seen the emergence of mul:ple CDNs (Akamai, Level 3, Google, Limelight, plus Amazon, Nejlix, and others in the making) which have further diluted any a3empt to monopolize the backbone. Also over the past 20+ years, we've seen an explosion in the number of buildings where some kind of peering takes place. In short, no one has been able to monopolize the Internet backbone. Now were seeing the emergence of mul:-lateral peering and even more backbone par:cipants.

31

The Internet backbone is a very interes:ng phenomenon. Its essen:ally unregulated. IANA (the body that supervises the assignment of addresses and other protocol number assignments) provides only coordina:on. If IANA withheld or manipulated assignments, their func:on could be quickly and informally bypassed.

Recently weve heard a lot about regula:ng the Internet especially since the revela:ons of NSA spying. But most such discussion is happening without any understanding of how the Internet backbone actually works.

Forecasts are iy, but the current system is extremely successful and extremely robust, so I am op:mis:c the Internet will con:nue to grow, indenitely.

32

33