Inter-domain Routing Protocol

2007

2

Overview

An Introduction to BGP BGP and the Stable Paths problem Convergence of BGP in the real world The End-to-End Effects of Internet Path

Selection

3

The Internet: Zooming In

ASes: Independently owned & operated commercial entities

CMU(PSC)

Comcast

Abilene

AT&T Cogent

Autonomous Systems (ASes)BGP

IGPs

(OSPF, etc)All ASes are not equal

4

Internet Routing

Internet organized as a two level hierarchy First level – autonomous systems (AS’s)

- AS – region of network under a single administrative domain

AS’s run an intra-domain routing protocols

- Distance Vector, e.g., RIP

- Link State, e.g., OSPF Between AS’s runs inter-domain routing

protocols, e.g., Border Gateway Routing (BGP)

- De facto standard today, BGP-4

5

Inter-domain Routing basics

Internet is composed of over 20000 AS’s

- All AS’s must cooperate to ensure reachability BGP = Border Gateway Protocol

- Policy-Based routing protocol

- De facto inter-domain routing protocol of Internet

Relatively simple protocol, but…

- Extremely complex configuration flexibility: entire world can see, and be impacted by, configuration mistakes

Many open research problems in policy, scalability, failover, configuration, correctness, security.

6

BGP Basics

Use TCP - AS’s exchange

Border Gateway Protocol (BGP), based on Bellman-Ford path vector

AS’s exchange reachability information through their BGP routers, only when routes change

BGP routing information – a sequence of AS’s indicating the path traversed by a route

General operations of a BGP router:- Learns multiple paths

- Picks best path according to its AS policies

- Install best pick in IP forwarding tables

7

BGP Operations (Simplified)

Establish session on TCP port 179

Exchange all active routes

Exchange incremental updates

AS1

AS2

While connection is ALIVE exchangeroute UPDATE messages

BGP session

8

Tier-1 ISPs have “default-free” routing tables (i.e., they don’t have any default routes), and typically have global reachability information. There are a handful of these today (about five or so).

Tier 1 ISP Tier 1 ISP

Tier 2Tier 2Tier 2Tier 2

Tier 1 ISP

Tier 1 ISP

Tier 2Tier 2

Tier 3

Tier 3Tier 2: Regional/National Tier 3: Local

Various Types of ISPs

9

AS relationships

Very complex economic landscape. Simplifying a bit:

- Transit: “I pay you to carry my packets to everywhere” (provider-customer)

- Peering: “For free, I carry your packets to my customers only.” (peer-peer)

Technical defn of tier-1 ISP: In the “default-free” zone. No transit.

- Note that other “tiers” are marketing, but convenient. “Tier 3” may connect to tier-1.

10

Customers and Providers

Customer pays provider for access to the Internet

provider

customer

IP trafficprovider customer

11

The “Peering” Relationship

peer peer

customerprovider

Peers provide transit between their respective customers

Peers do not provide transit between peers

Peers (often) do not exchange $$$trafficallowed

traffic NOTallowed

12

Peering Provides Shortcuts

Peering also allows connectivity betweenthe customers of “Tier 1” providers.

peer peer

customerprovider

13

Peering Wars

Reduces upstream transit costs

Can increase end-to-end performance

May be the only way to connect your customers to some part of the Internet (“Tier 1”)

You would rather have customers

Peers are usually your competitors

Peering relationships may require periodic renegotiation

Peering struggles are by far the most contentious issues in the ISP world!

Peering agreements are often confidential.

Peer Don’t Peer

14

Architecture of Dynamic Routing

AS 1

AS 2

BGP

EGP = Exterior Gateway Protocol

IGP = Interior Gateway Protocol

Metric based: OSPF, IS-IS, RIP, EIGRP (cisco)

Policy based: BGP

OSPF

EIGRP

15

AS-Path

Sequence of AS’s a route traverses Used for loop detection and to apply policy

120.10.0.0/16130.10.0.0/16

110.10.0.0/16

AS-1

AS-2

AS-3 AS-4

AS-5

120.10.0.0/16 AS-2 AS-3 AS-4

130.10.0.0/16 AS-2 AS-3

110.10.0.0/16 AS-2 AS-5

16

Four Types of BGP Messages

Open : Establish a peering session.

Keep Alive : Handshake at regular intervals.

Notification : Shuts down a peering session.

Update : Announcing new routes or

withdrawing previously announced routes.

announcement = prefix + attributes values

17

Two Types of BGP Neighbor Relationships

• External Neighbor (eBGP) in a different Autonomous Systems

• Internal Neighbor (iBGP) in the same Autonomous System

AS1

AS2

eBGP

iBGP

iBGP is routed using Interior Gateway Protocol (IGP) such as OSPF/IS-IS

18

iBGP Peers Must be Fully Meshed

iBGP neighbors do not announce routes received via iBGP to other iBGP neighbors.

eBGP update

iBGP updates

• iBGP is needed to avoid routing loops within an AS

• Injecting external routes into IGP does not scale and causes BGP policy information to be lost

• BGP does not provide “shortest path” routing

19

Important BGP attributes

Local PREF

- Local preference policy to choose “most” preferred route

Multi-exit Discriminator (MED)

- Which peering point to choose? Import Rules

- What route advertisements do I accept? Export Rules

- Which routes do I forward to whom?

20

Route Selection Summary

Highest Local Preference

Shortest ASPATH

Lowest MED

i-BGP < e-BGP

Lowest IGP cost to BGP egress

Lowest router ID

Traffic engineering

Enforce relationships

Throw up hands andbreak ties

22

Import Routes

Frompeer

Frompeer

Fromprovider

Fromprovider

From customer

From customer

provider route customer routepeer route ISP route

23

Export Routes

Topeer

Topeer

Tocustomer

Tocustomer

Toprovider

From provider

provider route customer routepeer route ISP route

filtersblock

24

Overview

An Introduction to BGP BGP and the Stable Paths problem Convergence of BGP in the real world The End-to-End Effects of Internet Path

Selection

25

aid in the design of policy analysis algorithms& heuristics aid in the analysis and design of BGP and extensions help explain some BGP routing anomalies provide a fun way of thinking about the protocol Our focus

What Problem is BGP solving?

Underlying problem

Shortest Paths

Distributed means of computing a solution

X?

RIP, OSPF, IS-IS

BGP

Having an X can

26

1

Q : How simple can X get? A: The Stable Paths Problem (SPP)

2 5 5 2 1 0

0

2 1 02 0

1 3 01 0

3 0

4 2 04 3 0

3

4

2

1

graph of nodes and edges node 0, called the origin for each non-zero node, a

set or permitted paths to the origin—this set always contains the “null path”

ranking of permitted paths at each node—null path is always least preferred

An instance of the SPP :

When modeling BGP : nodes represent BGP speaking border routers, and 0 represents a node originating some address block

most preferred…least preferred (not null)

27

5 5 2 1 0

1

A Solution to a Stable Paths Problem

2

0

2 1 02 0

1 3 01 0

3 0

4 2 04 3 0

3

4

2

1

node u’s assigned path is either the null path or is a path uwP, where wP is assigned to node w and {u,w} is an edge

in the graph,

each node is assigned the highest ranked path among those consistent with the paths assigned to its neighbors.

A Solution need not represent a shortest path tree, or a spanning tree.

A solution is an assignment of permitted paths to each node such that

28

Example: SHORTEST1

1 2

4

3

0

2 02 1 0

1 01 3 0

4 3 04 2 0

3 0

29

Example: SHORTEST1 (Solution)

1 2

4

3

0

2 02 1 0

1 01 3 0

4 3 04 2 0

3 0

30

Example: SHORTEST2

1 2

4

3

0

2 02 1 0

1 01 3 0

4 2 04 3 0

3 0

31

Example: SHORTEST2 (Solution)

1 2

4

3

0

2 02 1 0

1 01 3 0

4 2 04 3 0

3 0

32

Example: GOOD GADGET

1 2

4

3

0

2 1 02 0

1 3 01 0

4 3 04 2 0

3 0

33

Example: GOOD GADGET (Solution)

1 2

4

3

0

2 1 02 0

1 3 01 0

4 3 04 2 0

3 0 • No another solution:• 1 chooses (1 0)• 2 chooses (2 1 0)• 3 chooses (3 0)• 4 chooses (4 3 0)

34

A Stable Paths Problem may have multiple solutions

First solution

1

0

2

1 2 01 0

1

0

2

1

0

2

2 1 02 0

1 2 01 0

2 1 02 0

1 2 01 0

2 1 02 0

Second solutionDISAGREE

35

Example: NAUGHTY GADGET

1 2

4

3

0

2 1 02 0

1 3 01 0

4 3 04 2 0

3 4 2 03 0

36

Example: NAUGHTY GADGET (Solution 1)

1 2

4

3

0

2 1 02 0

1 3 01 0

4 3 04 2 0

3 4 2 03 0

37

Example: NAUGHTY GADGET (Solution 2)

1 2

4

3

0

2 1 02 0

1 3 01 0

4 3 04 2 0

3 4 2 03 0

38

SPP explains possibility of BGP divergence

Solvable Can Diverge

must converge must diverge

The SPP view :

BGP is not guaranteed to converge to a stable routing. Policy inconsistencies can lead to “livelock” protocol oscillations.

See “Persistent Route Oscillations in Inter-domain Routing” by K. Varadhan, etc. ISI report, 1996

39

Example: NAUGHTY GADGET

1 2

4

3

0

2 1 02 0

1 3 01 0

4 3 04 2 0

3 4 2 03 0

40

Example: BAD GADGET

1 2

4

3

0

2 1 02 0

1 3 01 0

4 2 04 3 0

3 4 2 03 0

That was one round of oscillation!

41

Example: BAD GADGET

1 2

4

3

0

2 1 02 0

1 3 01 0

4 2 04 3 0

3 4 2 03 0

• 2 chooses (2 0)• 4 chooses (4 2 0)• 3 chooses (3 4 2 0)• 1 chooses (1 0)

42

Example: BAD GADGET

1 2

4

3

0

2 1 02 0

1 3 01 0

4 2 04 3 0

3 4 2 03 0

… 2 chooses (2 1 0) …

• 2 chooses (2 0)• 4 chooses (4 2 0)• 3 chooses (3 4 2 0)• 1 chooses (1 0)

43

Example: BAD GADGET

1 2

4

3

0

2 1 02 0

1 3 01 0

4 2 04 3 0

3 4 2 03 0

… 4 chooses ( ) …

• 2 chooses (2 0)• 4 chooses (4 2 0)• 3 chooses (3 4 2 0)• 1 chooses (1 0)

44

Example: BAD GADGET

1 2

4

3

0

2 1 02 0

1 3 01 0

4 2 04 3 0

3 4 2 03 0

… 3 chooses (3 0) …

• 2 chooses (2 0)• 4 chooses (4 2 0)• 3 chooses (3 4 2 0)• 1 chooses (1 0)

45

Example: BAD GADGET

1 2

4

3

0

2 1 02 0

1 3 01 0

4 2 04 3 0

3 4 2 03 0

… 4 chooses (4 3 0) …

• 2 chooses (2 0)• 4 chooses (4 2 0)• 3 chooses (3 4 2 0)• 1 chooses (1 0)

46

Example: BAD GADGET

1 2

4

3

0

2 1 02 0

1 3 01 0

4 2 04 3 0

3 4 2 03 0

… 1 chooses (1 3 0) …

• 2 chooses (2 0)• 4 chooses (4 2 0)• 3 chooses (3 4 2 0)• 1 chooses (1 0)

47

Example: BAD GADGET

1 2

3

0

2 1 02 0

1 3 01 0

3 4 2 03 0

… 2 chooses (2 0) …

44 2 04 3 0

• 2 chooses (2 0)• 4 chooses (4 2 0)• 3 chooses (3 4 2 0)• 1 chooses (1 0)

48

Example: BAD GADGET

1 2

3

0

2 1 02 0

1 3 01 0

3 4 2 03 0

… 4 chooses (4 2 0) …

44 2 04 3 0

• 2 chooses (2 0)• 4 chooses (4 2 0)• 3 chooses (3 4 2 0)• 1 chooses (1 0)

49

Example: BAD GADGET

1 2

3

0

2 1 02 0

1 3 01 0

3 4 2 03 0

… 3 chooses (3 4 2 0) …

44 2 04 3 0

• 2 chooses (2 0)• 4 chooses (4 2 0)• 3 chooses (3 4 2 0)• 1 chooses (1 0)

That was one round of oscillation!

50

Example: BAD GADGET

1 2

4

3

0

2 1 02 0

1 3 01 0

4 2 04 3 0

3 4 2 03 0

• 2 chooses (2 0)• 4 chooses (4 2 0)• 3 chooses (3 4 2 0)• 1 chooses (1 0)

… 1 chooses (1 0) …

That was one round of oscillation!

51

1 2

4

3

0

2 1 02 0

1 3 01 0

4 2 04 3 0

3 4 2 03 0Result:

persistent oscillation

BAD GADGET : No Solution

In a BGP-like protocol• each node local decisions• at least one node can always improve its path

52

SURPRISE : Beware of Backup Policies

BGP is not robust :

it is not

guaranteed

to recover from

network failures.

1 2

3

0

2 1 02 0

1 3 01 0

3 4 2 03 0

4

4 04 2 04 3 0

Becomes BAD GADGETif link (4, 0) goes down !

53

PRECARIOUS

1

0

2

1 2 01 0

2 1 02 0

3

4

5 6

5 3 1 05 6 3 1 2 05 3 1 2 0

6 3 1 06 4 3 1 2 06 3 1 2 0

4 3 1 04 5 3 1 2 04 3 1 2 0

3 1 03 1 2 0

As with DISAGREE, this part has two distinct solutions

This part has a solution only when node 1 is assigned the direct path (1 0).

Has a solution, but can get “trapped”

54

Theoretical Results

The problem of determining whether an instance of stable paths problem is solvable is NP-complete

Shortest path route selection is provably safe

55

What is to be done?

StaticApproach

Inter-AScoordinationAutomated Analysis

of Routing Policies(This is very hard).

Dynamic Approach

Extend BGP with a dynamic means of detecting and suppressingpolicy-based oscillations?

These approaches are complementary

56

Overview

An Introduction to BGP BGP and the Stable Paths problem Convergence of BGP in the real world The End-to-End Effects of Internet Path

Selection

57

Convergence in the real-world?

[Labovitz99] Experimental results from two year study which measured 150,000 BGP faults injected into peering sessions at several ISPs

Found:

- Internet averages 3 minutes to converge after failover

- Some multihomed failovers (short to long ASPath) require 15 minutes

58

BGP Convergence Example

R

AS0 AS1

AS2AS3

*B R via AS3 B R via AS0,AS3 B R via AS2,AS3

*B R via AS3 B R via AS0,AS3 B R via AS2,AS3

*B R via AS3 B R via AS1,AS3 B R via AS2,AS3

AS0 AS1 AS2

** **B R via 203

*B R via 013 B R via 103

59

Convergence Result

If we assume

1. unbounded delay on BGP processing and propagation

2. Full BGP mesh BGP peers

3. Constrained shortest path first selection algorithm

Convergence time of BGP is O(N!), where N number of BGP speakers

There exists possible ordering of messages such that BGP will explore all possible ASPaths of all possible lengths

60

Overview

An Introduction to BGP BGP and the Stable Paths problem Convergence of BGP in the real world The End-to-End Effects of Internet Path

Selection

61

End-to-end effects of Path Selection

Goal of study: Quantify and understand the impact of path selection on end-to-end performance

Basic metric

- Let X = performance of default path

- Let Y = performance of best path

- Y-X = cost of using default path Technical issues

- How to find the best path?

- How to measure the best path?

62

Approximating the best path

Key Idea

- Use end-to-end measurements to extrapolate potential alternate paths

Rough Approach

- Measure paths between pairs of hosts

- Generate synthetic topology – full NxN mesh

- Conservative approximation of best path Question: Given a selection of N hosts, how

crude is this approximation?

63

Methodology

For each pair of end-hosts, calculate:

- Average round-trip time

- Average loss rate

- Average bandwidth Generate synthetic alternate paths (based

on long-term averages) For each pair of hosts, graph difference

between default path and alternate path

64

Courtesy: Stefan Savage

65

Courtesy:

Stefan Savage

66

Courtesy:

Stefan Savage

67

Courtesy:

Stefan Savage

68

Why Path Selection is imperfect?

Technical Reasons

- Single path routing

- Non-topological route aggregation

- Coarse routing metrics (AS_PATH)

- Local policy decisions Economic Reasons

- Disincentive to offer transit

- Minimal incentive to optimize transit traffic