An Analysis of BGP Multiple Origin AS (MOAS)Conflicts

Xiaoliang Zhao, Dan Pei, Lan Wang, Dan Massey, Allison Mankin, S. Felix Wu,Lixia Zhang

Abstract-This paper presents a detailed study of BGP Multiple Ori-

gin AS (MOAS) conflicts observed in the Internet. A MOASconllict occurs when a particular prefix appears to originatefrom more than one AS. We analyzed data from archivedBGP routing tables over 1279 days. Most of the conflictswere short-lived, lasting only a small number of days. Thepotential causes for the MOAS conflicts and impact on BGPfault-tolerance are discussed in detail.

I. INTRODUCTION

This paper presents a detailed analysis of BCP[12]routes that appear to originate from multiple AutonomousSystems. The Internet is made of thousands of Au-tonomous Systems, loosely defined as a connected groupof one or more IP prefixes which have a single and clearlydefined routing policy[l 11. BGP[ 121 is the standard inter-AS routing protocol. A BGP route lists a particular pre-fix (destination) and the path of ASes used to reach thatprefix. The last AS in an AS path should be the originof the BGP routes. A Multiple Origin Autonomous Sys-tem (MOAS) conflict occurs if a prefix appears to originatefrom more than one ASes. More precisely, suppose prefixd is associated with AS paths asp1 = (pl,p2,. . .pn) and

asp2 = (ql,q2,. . . qm). We say a MOAS conflict occursifxh # Qm-

In an effort to improve the fault-tolerance and secu-rity properties of the BGP routing protocol, we have beenmeasuring the behavior of BGP. The MOAS conflicts areinteresting to us for a number of reasons. First, RFC

This material is based upon work supported by the Defense AdvancedResearch Projects Agency (DARPA) under Contract No DABT63-OO-C-1027. Any opinions, findings and conclusions or recommendationsexpressed in this material are those of the authors and do not necessarilyreflect the views of the DARPA.

xzhao@unity.ncsu.edu, peidan@cs.ucla.edu, lanw@cs.ucla.edu,masseyd@isi.edu, mankin@isi.edu, wu@cs.ucdavis.edu,lixia@cs.ucla..edu

Permission to make digital or hard copies of all or part ofthis work forpersonal or classroom use is granted without fee provided that copiesare not made or distributed for profit or commercial advantage and thatcopies bear this notice and the full citation on the first page. To copyotherwise, or republish, to post on servers or to redistribute to lists,requires prior specific permission and/or a fee.IMW’OI, November l-2,2001, San Francisco. CA, USA.Copyright2001 ACM I-58113-435-5/01/0011...$5.00.

1930[ 1 l] recommends that a prefix should originate froma single AS, but MOAS conflict may occur for a limitednumber of valid reasons. Second, MOAS conflicts couldbe the result of a fault or an attack, where a BGP routerfalsely originates routes to some other organization’s pre-fixes. We would like to understand the characteristics ofvalid MOAS conflicts due to operational needs and invalidMOAS conflicts caused by faults.

MOAS conflict data was obtained from Internet routersand was analyzed based on the total number of conflicts,duration of the conflicts, and the prefix length. Both thenumber of MOAS conflicts and distribution of the dura-tion of MOAS conflicts were different than what we antic-ipated. This paper presents the measurement results andanalysis of potential causes.

The remainder of the paper is organized as follows. Sec-tion II reviews the related work. Section III describes themethodology we used to collect and process the data. Sec-tion IV presents the MOAS conflict data. Section V andVI provide detailed analysis of the results and our explana-tions of the results. Section VII discusses the implicationsof this work and summarizes the paper.

I I . RE L A T E D W O R K

MOAS conflicts have also been observed by other re-searchers, but no one has considered the problem in de-tail. The most relevant work comes from Geoff Huston’sBGP Table Statistics website[9]. Starting on 2/18/2OOl,this site began tracking a daily count of MOAS conflicts’using data from some ISPs and from the Oregon RouteViews Server. On 04/19/2001, the website switched totracking MOAS conflicts on a bi-hourly instead of dailybasis. However, the BGP Table Statistics work providesonly a basic count of MOAS conflicts and no further ex-planations or analysis is offered.

The MOAS conflict issue has also been discussed withinthe IETF. RFC 1930[ 1 l] recommends that a prefix shouldbelong to only one AS. If this recommendation was fol-lowed, MOAS conflicts would not occur, with the possibleexception of a few unique cases discussed further in Sec-tion VI-D. Berkowitz[ 131 discussed the potential causes of

‘Huston uses the term “multiple-origin prefixes” in place of our term“MOAS conf l i c t s”

3 1

MOAS conflicts. However, the discussion is not completeand no implications of MOAS conflicts are analyzed.

III . METHODOLOGY

The BGP route for a prefix (destination) includes an ASpath. The last AS along the path to the prefix is consideredto be the origin AS. We examined the AS paths that led tothe same prefix but ended in different origin ASes.

We primarily used data from the Oregon Route Viewsserver [S] to obtain the BGP routes and AS paths used inthis study. Currently, the Oregon Route Views server peerswith 54 BGP routers in 43 different ASes. Each peer ex-ports its BGP routing table to the Route Views server.

The Oregon Route Views data is particularly attractivebecause it provides data from a number of different van-tage points. The data obtained from a particular localpoint, such as in an individual ISP, may show a smallernumber of MOAS conflicts since fewer potential AS pathsmay be visible at that point in the network. For example, ata randomly selected time, the Oregon Route Views serverobserved 1364 MOAS conflicts, but three other individualISPs observed 30, 12, and 228 MOAS conflicts during thesame period. This only means that fewer MOAS conflictswere visible to these ISPs and even the number of MOASconflicts observed from the Oregon Route Views Servermay underestimate the total number of MOAS conflicts.

To obtain a relatively complete view, we used archivedOregon Route Views data from both NLANR[2] andPCHnet [3]. NLANR archived the Oregon Route Viewsdata on a daily basis from 1 l/08/1998 to 03/16/2001.PCH.net archived the Oregon Route Views data on a dailybasis from 03/16/2001 to the present. The MOAS conflictsare identified by prefixes only no matter whether a MOASconflict was conflicted by same set of origin ASes or theconflict was continuous.

Note that AS sets did not play any meaningful role in ourstudy. An AS path typically consists of the sequences ofAS numbers used to reach prefix, but due to factors such asaggregation, the AS path may also contain AS sets as wellas AS sequences. Out of of over 1OOK prefixes observed,roughly 12 routes ended in AS sets and these 12 routeswere not included in the study.

IV RESULTS

The total number and durations of MOAS conflicts de-viated substantially from our expectation. Based on theseresults, we believe the nature of these conflicts differs fromwhat one might expect based on documents such as [ 111.

Fig. 1. The number of MOAS conflicts from 1 l/1997 to 07/200

Median of MOAS conflicts

1 2001 ] 1294 1 36.1% I

Fig. 2. Median of MOAS conflicts per year

A . Total Number of MOAS ConJlicts

Figure 1 shows the total number of conflicts from1 l/08/1997 to 07/18/2001 2. Overall 38225 conflicts wereobserved over 1279 days. The median number of MOASconflicts for each year are listed in Figure 2. There is anincrease from 683 conflicts in 1998 to 1294 conflicts in2001.

B. Duration of MOAS Conjlicts

Figure 3 shows the duration of MOAS conflicts, basedon the data (Figure 1). Figure 3 shows that most of theconflicts are short-lived. 13730 out of 38225 conflicts ap-peared only once and lasted less than one day. 11358 ofthese one-time conflicts can be attributed to a configura-tion fault that occurred on April 7th, 1998. Excluding theone-time conflicts, the expectation of the duration is 30.9days. Taking into account that many other short-lived con-flicts might also be due to faults, we considered the dataset which contains only conflicts whose duration is greaterthan 9 days (a total of 10177 conflicts). For these con-flicts, the expectation of the duration is 107.5 days with1002 conflicts lasted longer than 300 days. Figure 4 liststhe expectation of the duration from the different data sets.The longest duration was 1246 days out of a possible 1279days and 1326 conflicts were still ongoing as of the datethe paper was written.

The duration of an individual conflict counts the totalnumber days of the conflict was in existence, regardless of

‘The number of conflicts reached its peaks of 11842 on 04/07/1998and 10226 on 04/06/2001.

3 2

Fig. 3. Duration of MOAS conflicts

Expectation (days) Measured data set30.9 longer than 0 day47.1 lotwer than 1 days

I 107.5 / lower than 9 days175.3 1longer than 2 9 days281.8 1longer than 8 9 days

Fig. 4. Expectation of the duration of MOAS conflicts

whether the conflict was continuous and whether the sameASes were involved.

The results seem a little surprising if one assumes thatmulti-homing, discussed in Section VI-B, is the major rea-son for the MOAS conflicts. Multi-homing would seemto imply that the MOAS conflicts should last longer thanwhat is observed here and this is discussed further in Sec-tion VI-F.

C. Disttibution of MO.45’ Cor1flict.sFigure 5 shows the distribution of conflicts among prefix

length. The /24 (netmask of 255.255.255.0) attracts mostof conflicts. This is not unexpected since /24 prefixes makeup the bulk of the BGP routing table.

Fig. 5. Distribution among prefix length

Fig. 6. Distribution of classes

V. CLASSIFICATION OFMOASCON~ICTS

If a MOAS conflict occurs, prefix p will be associatedwith at least two different AS paths:

l asp1 = bl,PZ,. . .PIl)l asp2 = kl, 921.. . sm)

By definition, p, # q,,, for a MOAS conflicts. In orderto better understand the type of conflicts and the potentialcauses, we divided the MOAS conflicts into three classesbased on relationships between the two AS paths.OiigTrmAS: p, = qj (j < m).In this case, AS p, announces itself as the origin AS inasp1 and announces itself as a transit AS in aspz.SplitVkW: pi = qj (i < ?Z,j < m).In this case, AS pi announces different routes to differentneighbors.DistinctPaths. pi # qj (Vi E [l..n],j E [l..m]).In this case, there are two totally different routes for theprefix d.

Instances of all three cases were observed and Figure6 shows the number of conflicts for each class. In theOrigTranAS class, an AS acts as both the origin AS anda transit AS. In the SplitView class, a transit AS offers twodifferent paths to the prefix and these paths end in differentorigin ASes.

The OrigTranAS and SplitView conflicts indicate that asingle AS may advertise multiple paths to the same pre-fix. This is often because of the traffic engineering prac-tices used at large ISPs. An AS might prefer that trafficto the same destination flow through different paths due toconstraints such as geographical distances, link speed, oreconomic reasons.

In the DistinctPaths class, there are two completely dis-joint AS paths for the same prefix. Figure 6 shows thatthe DistinctPaths class is dominant in the MOAS conflicts,which is not unexpected because BGP only choose onebest route if no traffic engineering practice.

33

VI. EXPLANATIONS ANDIMPLICATIONS

There are a number of possible explanations for MOASconflicts. Unique cases such as exchange points, someforms of multi-homing, and faults all contribute to theMOAS conflicts. Each of these factors was observed in

this study.

A. Exchange Point Addresses

One potential cause of MOAS conflicts involves theprefixes associated with exchange points (or equivalently,links connecting ASes). A prefix associated with an ex-change point is directly reachable from all the ASes at theexchange point and each AS at the exchange point mightadvertise the prefix as if it comes directly from that AS.

However, exchange point prefixes make up a small per-centage of the MOAS conflicts observed in this study. Inthe examined BGP data, 30 out of 38225 prefixes couldbe definitively identified as exchange point prefixes. Ouranalysis of exchange point prefixes may underestimate thetotal number of exchange point prefixes, but the numberof exchange point prefixes remains relatively small evenif our estimate is off by two orders of magnitude. All ofthese exchange point prefix conflicts lasted for long pe-riods, consisting of most or all of the observation periods.These MOAS conflicts do not present a problem for packetforwarding since each AS originating the route can directlyreach the prefix.

B. Multi-homing Without BGP

In some cases, multi-homing can occur without the useof BGP and this can result in MOAS conflicts. Supposethere is a link between two ASes, but the routing acrossthis link does not use BGP (and instead relies on staticrouting or some IGP). From a BGP perspective, it appearsas if one AS can directly reach prefixes belonging to theother AS.

Again one would expect these conflicts to be long last-ing since static routes are likely to have a long lifetime.These MOAS conflicts could present a problem for packetforwarding if the links necessary to support the staticroutes fail.

C. Multi-horning with Private AS Numbers

To prevent AS number exhaustion, Haas [lo] suggeststhat a multi-homed customer uses a private AS numberwhich is mutually agreeable to all providers. This tech-nique is called AS number Substitution on Egress (ASE).If deployed, this approach could produce MOAS conflictsbecause the private AS number should be stripped off by

the upstream providers and the real origin information willbe lost.

Based on discussions with network operators, we do notbelieve this technique is used widely in practice. TheseMOAS conflicts would not present a problem for packetforwarding since all upstream providers can reach the pri-vate AS. Furthermore, if the link to the private AS is lost,the corresponding BGP route will also be withdrawn.

Because the links using non-BGP routing mechanismsor private AS numbers are “hidden” to BGP, the pure BGPdata can not tell whether or not a MOAS conflict is due tomulti-homing without BGP or multi-homing with privateAS number. However, by contacting individual ASes, wedid confirm such occurrences.

D. Theoretical Causes

Other factors have the potential to cause MOAS con-flicts, but these factors did not occur during our study. Inparticular, RFC 1930[ 1 l] notes that aggregation could re-sult in routes that end in AS sets. But overall, we typicallyobserved 12 prefixes which ended in AS sets and these ASsets were consistent with each other.

Anycast address would also create MOAS conflictssince an anycast prefix is intended to originate from mul-tiple ASes. No prefixes in our study were identified asanycast addresses.

E. Faulty or Malicious ConjigurationsMOAS conflicts can also occur when an AS incorrectly

originates routes to some other organization’s prefixes.This could occur due to configuration errors or even inten-tional attacks. Often, the faulty AS does not have a route tothe incorrectly originated prefixes and packets that use theincorrectly originated route will reach the faulty AS andthen be lost.

Figure 1 shows several notable examples of MOAS con-flicts caused by faults. The graph shows a large spike onApril 7th 1998 and AS 8584 was involved in 11357 out of11842 conflicts that occurred during that day. Discussionson a network operators mailing list[4] indicated that AS8584 falsely originated routes to those conflicted prefixes.Consequently, some ASes selected the incorrectly origi-nated route. Packets sent along this incorrectly originatedroute would reach AS 8584 and would then be lost.

The graph also shows a large spike on April lOth, 2001and the sequence (AS 3561, AS 15412) was involved in5532 out of 6627 MOAS conflicts that occurred during thatday. Based on the archived data from RIPE RIS [l], AS15412 normally originates only 5 prefixes. However, onApril 6th, AS 15412 suddenly originated thousands pre-fixes due to a configuration error[5].

34

On April 25th, 1997, a severe Internet outage oc-curred[7] when one ISP falsely de-aggregated most ofthe Internet routing table and advertised the prefixes as ifthey originated from the faulty ISP[6]. The falsely origi-nated prefixes resulted in MOAS conflicts. These exam-

ples show that invalid MOAS conflicts do occur and canhave serious impacts on Internet routing.

Faulty aggregation could also cause MOAS conflicts. Infaulty aggregation, an AS advertises an aggregated prefix,even though some of more specific prefixes are not reach-able by the AS. A MOAS conflict occurs if an aggregateroute is also generated by some other AS. Packets that usethe faulty aggregated route will travel to the faulty AS andthen may not be able to reach all the more specific prefixes.

F: MOAS Conflict Durations and Potential Causes

With the exception of faults and intentional attacks, thepossible explanations should have created long durationMOAS conflicts. MOAS conflicts for exchange point pre-fixes should remain as long as two or more ASes chooseto advertise a route to the exchange point. The data con-firmed this expected pattern and exchange point MOASconflicts persisted for most, if not all, of the study. Multi-homing without BGP and multi-homing with Private ASnumbers both require router policy configurations at twoor more ASes and the resulting MOAS conflicts shouldpersist for as long as the multi-homing policy remains inplace. We expected that multi-homing policies (and theresulting MOAS conflicts) would occur over months, notdays. But the data in Section IV shows a large number ofshort duration conflicts.

One possible reason for short-lived MOAS conflicts isthat MOAS conflicts could occur during a transition periodwhen a non-BGP customer switches from one provider toanother. To guarantee the connectivity to the non-BGPcustomer, it is possible for both providers to originate thecustomer’s prefix for a short period. Another possible andmore likely reason for short-lived MOAS conflicts is routermis-configurations or other faults. These conflicts disap-pear when the faults are detected and corrected.

Overall, the duration can be a useful heuristic to dis-tinguish between valid MOAS conflicts and invalid ones.However, such differentiation can not be accurate enoughto be a solution to validate MOAS conflicts.

VII. SUMMARY

The MOAS data presented in this paper can help in un-derstanding the operational behavior of BGP. At a mini-mum, one would like to know if types of MOAS conflictsexpected to occur actually match the type of conflicts actu-ally occurring in the Internet. These results would indicate

there are a large number of faults or large number of veryshort lived multi-homing policies.

From the standpoint of fault-tolerance and security,MOAS conflicts pose an interesting challenge. On theone hand, MOAS conflicts can occur for valid reasons,such as multi-homing without BGP and advertising routesto exchange points. On the other hand, router mis-configurations have also produced MOAS conflicts. Largescale network outages and other problems have been as-sociated with MOAS conflicts. When a MOAS conflict isobserved, we would like to be able to determine whetherit is the result of a fault or a valid change in routing/multi-homing policy. Based on this MOAS data alone, we cannot accurately differentiate a fault from a valid policychange, but we can utilize the MOAS analysis results asa valuable input to address BGP problems and we are in-vestigating techniques for identifying invalid conflicts witha high degree of certainty.

VIII. A C K N O W L E D G M E N T S

We would like to thank a number of network opera-tors and BGP routing experts for the advice. In particular,Randy Bush and Geoff Huston provided useful insights onthe operation of large ISPs. Also, we would like to thankanonymous reviewers for their valuable comments.

HI

I21

[31

[41

[51

161

[71

PI

[91

WI

illI

WI

v31

R E F E R E N C E S

RIPE Routing Information Service,http://www.ripe.net/ripencc/pub-services/np/ris-index.htmlNational Laboratory for Applied Network Research,http://moat.nlanr.net/Routing/rawdaWPCH.ne t , http://www.pch.netJdocuments/datafrouting-tables/route-views.oregon-ix.netlB. Kroenung, “AS8584 taking over the internet”, NANOGmailing list, msg00047, Apr. 7, 1998.J. Farrar, “C&W routing instability”, NANOG mailing list,msg00209, Apr. 6,200l.V. J. Bono, “7007 Explanation and Apology”, NANOG mail-ing list, msgOO444, Apr. 26, 1997.R. Barrett et al, “Routing Snafu Causes In-ternet Outage”, ZDNet, April 25, 1997.http:ilwww.zdnet.com/zdnn/content/inwol~25/inwo0009.htmlUniversi ty of Oregon R o u t e Views Project,http://www.antc.uoregon.edulroute-views/G. Huston, “BGP table statistics”,http://www.telstra.net/ops/bgp/as6447/bgp-multi-orgas.html.J . Haas, “Autonomous System Number Substitution onEgress”, Internet Draft, Working in Progress, 2001.J. Hawkinson and T. Bates, “Guidelines for creation, selection,and registration of an Autonomous System (AS)“, RFC 1930.1996.Y. Rekhter and T. Li, “A Border Gateway Protocol 4 (BGP-4)‘,RFC 1771. 1995.H.Berkowitz, E.Davies and L. Andersson, “An ExperimentalMethodology for Analysis of Growth in the Global RoutingTable”, Internet Draft, Working in Progress, July, 2001.

35