Fast Accurate Computation of Large-Scale IP TrafficMatrices from Link Loads

Yin Zhang Matthew Roughan Nick Duffield Albert GreenbergAT&T Labs— Research, Shannon Laboratory

180 Park Avenue, Florham Park, NJ 07932�yzhang,roughan,duffield,albert� @research.att.com

ABSTRACTA matrix giving the traffic volumesbetweenorigin anddestinationin a network hastremendouslypotentialutility for network capacityplanningandmanagement.Unfortunately, traffic matricesaregener-ally unavailablein largeoperationalIP networks. On theotherhand,link loadmeasurementsarereadilyavailablein IP networks. In thispaper, we proposea new methodfor practicaland rapid inferenceof traffic matricesin IP networksfrom link loadmeasurements,aug-mentedby readilyavailablenetwork androutingconfigurationinfor-mation. We applyandvalidatethemethodby computingbackbone-routerto backbone-routertraffic matricesonalargeoperationaltier-1IP network – a problemanorderof magnitudelarger thanany othercomparablemethodhastackled. The resultsshow that the methodis remarkablyfastandaccurate,deliveringthetraffic matrix in underfiveseconds.

Categoriesand SubjectDescriptorsC.2.3[Computer-CommunicationsNetworks]: NetworkOperations—networkmonitoring

GeneralTermsmeasurement,performance

KeywordsTraffic Matrix Estimation,Traffic Engineering,SNMP

1. INTRODUCTIONA fundamentalobstacleto developingsoundmethodsfor network

andtraffic engineeringin operationalIP networkstodayis theinabil-ity of network operatorsto measurethetraffic matrix. A traffic matrixprovides,for every ingresspoint � into thenetwork andegresspoint�

out of the network, the volume of traffic ����� from � to�

over agiven time interval. Taken togetherwith network topology, routingandfault data,the traffic matrix canprovide a greatdealof help inthediagnosisandmanagementof network congestion[1]. On longertime scales,traffic matricesarecritical inputsto network design,ca-pacityplanningandbusinessplanning.

Unfortunately, today’sproductionsystemsfor IPnetworkmeasure-mentdo not provide the inputsneededfor direct computationof IPtraffic matrices.Instead,thesesystemsgatherdataon:

Permissionto make digital or hard copiesof all or part of this work forpersonalor classroomuseis grantedwithout fee provided that copiesarenot madeor distributedfor profit or commercialadvantageandthatcopiesbearthisnoticeandthefull citationon thefirst page.To copy otherwise,torepublish,to postonserversor to redistributeto lists,requiresprior specificpermissionand/ora fee.SIGMETRICS’03,June10–14,2003,SanDiego,California,USA.Copyright 2003ACM 1-58113-664-1/03/0006...$5.00.

resourceutilizationatnetwork nodesandlinks, (e.g.link loads); end-to-endperformancemetricsfor specifictransactions,suchasoneway delaystatisticsfor packetsexchangedbetweenmeasure-mentserversat thenetwork edge; statusandconfigurationof network topologyandrouting.

Thoughthesemeasurementsmayreveal traffic anomaliesor conges-tion problems,they do not in generalrevealpotentialsolutions.Forinstance,link loadmeasurementsmayrevealcongestiononalink, butshedlittle light on its cause,which in generalrequiresunderstandingthetraffic matrix.

Theprincipal contribution of this paperis a simple,efficient, andaccuratemethodfor computingtraffic matrix estimatesfor IP net-works,from widely availabledata:link loadandnetwork routingandconfigurationdata.Themethoddrawson ideasfrom “gravity model-ing” [2, 3, 4, 5, 6, 7] and“tomographicmethods”[8, 9, 10,11,12]. Italsomakesuseof network configurationdatato dramaticallyreducecomputationalcomplexity.

We have validatedthe methodagainst direct traffic matrix mea-surementsfrom detailedflow level dataon an operationaltier-1 IPnetwork, andtheresultsshow very goodaccuracy. It alsohastheap-pealingcharacteristicthatadditionalinformation,sayfrom flow leveltraces,may be includedin a straightforward manner. The methodis very fast, taking lessthan 5 secondson a 336 MHz Ultrasparc-II processorto computea backbone-routerto backbone-routertrafficmatrix on the tier-1 IP network. The method,andits gravity modelprior havebeenusedin thatnetwork since2001for avarietyof tasksrangingfrom traffic engineeringto router/link failureanalysisto ca-pacityplanning,with considerablesuccess.

At their simplest,gravity modelsarebasedon theassumptionof asimpleproportionalityrelationship[2, 4]:

����� ������� ��������� (1)

where����� � and ����� denotethetotal traffic enteringthenetwork at �andexiting at

�, quantitiesthatcanbeobtainedby summinglink load

dataat the network edge. (SeeSection3.1.) The gravity modelas-sumesthatthetraffic componentfrom or to agivensitedependsonlythe total traffic enteringor leaving thatsite. By appropriatenormal-ization,thegravity modelsolutionis guaranteedto beconsistentwithmeasuredlink loadsat thenetwork edge,but notnecessarilysoin theinterior links. Alternatively, tomographicmethodsarebasedon thesystemof linearequations:

������� (2)

where � is the traffic matrix (written asa columnvector), � repre-sentslink loads,and � thenetwork routingmatrix – seeSection3.2for details.In essence,equation(2) statesthatthetraffic matrix mustbeconsistentwith network routingandmeasuredlink loadsthrough-out the network, not just at the edge.However, this matrix equalityis highly under-constrained,andso allows many solutions. Tomo-graphicmethodsdiffer in how a single “best” solution is identified

from the possibilities. The majority of existing statisticaltomog-raphy� approaches(commonlyreferredto as“network tomography”methods)usemodelsof the higher order statisticsof the link loaddatato createadditionalconstraints.In contrast,optimization-basedtomography approaches(e.g.,linearor quadraticprogramming)oftenattemptto find asolutionthatoptimizesanobjective function.

The methodintroducedin this paperrefinesand combinesbothgravity andtomographicmethods:

1. We solve the gravity model using edgelink load data. Ad-ditional informationon routing betweenpointsof ingressandegressfor traffic flowscanbeincorporatedto obtainsignificantimprovements.In thenumericalresultspresentedhere,we in-corporateinformationto modeltraffic exchangedwith peernet-works(Section3.1.2).

2. As the final estimateof the traffic matrix, we apply quadraticprogrammingto determinethe solution in the spaceof thoseadmittedby the tomography modelclosestto thesolutionob-tainedby thegravity model.Thissteputilizesall availablelinkload data,anddoesnot require(higher-order)statisticsor ad-ditional traffic modelingassumptions.Thekey computationalchallengeis to computethepseudo-inverseof theroutingma-trix , which hashigh dimensionality. To overcomethis chal-lenge,weapplynetwork configurationandroutingdatato dra-matically decreasethe problemdimension. Iterative propor-tional fitting is usedto ensurethenon-negativity of theresults.

We termthismethodfor computingIP traffic matricesthetomograv-ity method,for wantof abettername.

The validation of the tomogravity methodis basedon a set ofhourly traffic matricesderived from direct flow level measurementsusingthemethodologydescribedin [13]. Thesetraffic matricescoverover 2/3 of a tier-1 IP backbonenetwork (including all the peeringtraffic) over June2002. Obtainingdirect flow level measurementacrosslargeIP networkstodayis a far moretaxingandcomplex taskthanlink androuterconfigurationmeasurement,dueto limited andinconsistentroutersupportfor flow level measurementcapabilities.

Figure1 providesanindicationof theaccuracy of themethod.Themethodis remarkablyaccuratefor the all but the smallestentriesinthetraffic matrix. We notethat the largervaluesin thetraffic matrixdominatenetwork andtraffic engineeringapplications[14, 15, 13].Themajorityof thetraffic lieswithin ! 23%relativeerror, andamorethan30%of thematrixelementshavenegligible error.

A moredetailedexaminationof thedatawill show thattherelativeerrorsarelargestfor thesmallestmatrix elements,which fortunatelydo not have largeabsoluteerrorsin generalandarethusunlikely tomatterin applications. The matrix elementsof most importance–the largestvalues– arethe mostaccurate.Further, all of the errorswerefound to bewell-behaved, that is nonehave overly largeabso-lute errors,andthey vary smoothlyover time, meaningtheir impacton operationaltaskswill not be dramatic,even wherethe errorsarelargest.

Evenmoreimportantlyfrom anoperationsperspective,predictionsbasedontheestimatedtraffic matrixareremarkablyaccurate.For in-stance,therelative accuracy of computedlink loadsbasedon thees-timatedtraffic matrix arewithin a few percentof thereal link loads.Themethodinsuresthis will be thecase,but interestingly, otherre-sultssuchasthedistribution functionsfor thesizesof traffic matrixelementsbasedon real andestimateddataarealmostindistinguish-ableaswell.

Thepaperis organizedasfollows: we startin Section2 with basicnetwork conceptsand terminology, and the featuresof the datawehave available. This is followed by a detaileddescriptionof the to-mogravity methodin Section3. Thenext section(Section4) presentsour validation resultsof the method,basedon real network trafficmatrices.Finally weconcludethepaperin Section5.

−100 −50 0 50 1000

5

10

15

20

25

30

35

−23% +22%

% o

f flo

ws

relative error (%)

Figure 1: Relative errors of traffic matrix estimatescompared todir ectestimatesof the traffic matrix (fr om flow level data) for thelargestcomponentsof the traffic matrix (representingover 75%of the network traffic). Note that a significant proportion of theflows (more than 30%) have a negligible error. The two verticaldashedlinesshow the 5th and 95th percentilesof the distrib ution,showing that theselie within !�"�# %.

1.1 RelatedWorkTomographicmethodshave beenwidely andsuccessfullyapplied,

for example,in ComputerAided Tomography (CAT) scans,usedinmedical imaging. Thesemethodsdiffer in how they deal with theunder-determinationof the systemof tomographicconstraintequa-tions. Optimization-basedtomography approachestypically find asolution that optimizesan objective function, whereasnetwork to-mography approachesoftenusethehigherorderstatisticsof thelinkloaddatato createadditionalconstraints.

Vardi[8] first puttheideasof network tomography into practiceforcomputingtraffic matricesin communicationsnetworks,with subse-quentcontributionsby TebaldiandWest[9], andby Caoet al. [10].Thereis a dual of this problemalso referredto asnetwork tomog-raphy in which link performancemetricsaredeterminedfrom pathmeasurements[11, 12], but this doesnotdirectly concernushere.

Network tomography, in somesense,comprisesdeterminingthesolution � to equation(2), or at leastthe parametersof somemodelof � , from measurementsof � . As notedabove, this systemis highlyunder-constrained,andsothechallengeis to choosethe“best” solu-tion from the spaceof possibilities. The typical approachhasbeento useadditionalmodelingassumptionto derive constraintsfrom thehigherorderstatisticsof the traffic. For instance,Vardi [8], adopteda Poissonianmodelin which the $%� areindependentPoissonianran-domvariablesof mean&'� . He observed that thenthe &'� arePoisso-nian randomvariableswith covarianceCov ()&'�+*,&�.- � /10 �23� /�45/where 0 �23� / � � / / . Thus, the equations�6� 4 aresupple-mentedwith theequations7 �80 4 where9:�2 denotesthemeasuredcovarianceof thetraffic rateacrosslinks � and

�. We canwrite these

compactlyas

�7 � � ; �=< (3)

Vardi establishedthat underrealisticconditionson the matrix

;,

thePoissonrates� areidentifiable,in thesensethattwo differentsetsof rates� cannotgive rise to thesameasymptoticdistributionsof �and 7 for large numbersof probes. Caoet al. adopteda Gaussianmodelin which thevarianceof the

4 � hasa specifiedpower-law de-pendenceon themean.Themeanratesof this modelareidentifiableunderthesameconditionson

;.

Directly solving (3) for finitely many measurementsis problem-atic. Due to statisticalvariability of the � and 7 , the equationsaregenerallyinconsistent,while someof the 9:� maybenegative. To cir-cumvent theseproblems,Vardi employed an iterative approachthatusestheEM algorithm[16] to find approximatesolutions.In theap-proachof Caoetal., amodifiedEM algorithmis useddirectly to findmaximallikelihoodparametersin theGaussianmodel.Convergenceis hastenedby usingsecond-ordermethods.Even with suchmeth-ods, the complexity of the network tomography approachesgrowsas >?()@BA.- , where @ is the numberof regions,althougha reductionto >C()@CDE- is claimedfor topologiesin which theingress-egresspairscanbepartitionedovermostlydisjoint regions[17].

Furthermore,Medinaet al. [7] shows that the basicassumptionsunderlyingthestatisticalmodels(Poissonor Gaussian)arenot justi-fied, andthat themethodsabove mayperformbadlywhentheir un-derlyingassumptionsareviolated.Thepaperconcluded[7] thatnoneof the prior methodsfor computingtraffic matriceswassatisfactoryfor even a PoPto PoPtraffic matrix on a large network, let aloneaBR to BR matrix.

An alternative that is well known in socialsciencesfor modelingcommodityexchangesis thegravity model(See,for example[5, 6]).In network applications,gravity modelshavebeenusedto modelmo-bility in wirelessnetworks[3], andthevolumeof telephonecallsin anetwork [4]. Recently, variantsof gravity modelshavebeenproposedfor computingIP network traffic matrices[18, 7, 2]. For instance,[7]proposesan approachbasedon using the choice modelsto modelPoPfanouts, which canbeviewedasa variantof thegravity modelapproach.Thepaperalsosuggestsusingtheirmethodto generatepri-ors to serve asinputsto statisticaltomography techniques,but doesnot testthis idea. An alternative generalizationof thegravity model(detailedbelow) that explicitly modelsinter-peerrouting was usedby theauthorsof this currentpaperin capacityplanningexercisesin2001for anoperationalIP network.

As notedabove, gravity modelsaretypically basedon edgedata,andassuchdonotguaranteeconsistency with theobservedlink loadsontheinteriorof thenetwork. Of course,in their full generality, grav-ity modelscanindeedincludesuchinformationthroughtheuseof the@6F @ friction matrix for @ ingress/egresspoints(seeSection3.1fordetails).But the inferenceof thecompletefriction matrix is a prob-lem of the samecomplexity asinferenceof the traffic matrix itself.Wepreferto solve thelatterproblemdirectly.

Gravity models(basedon edgedata)are>?(G@CDE- complexity in theworst case,for @ ingress/egresspoints(aswe needto compute@�Delementsin thetraffic matrix),but thenumberof computationsis ac-tually rathersmallperterm,andsothesemethodsarefastoneventhelargestnetwork sizes.Hencegravity modelsarequiteappealing,butwe shalldemonstratethemodelof [2] canbesignificantlyimprovedby incorporatinginformationfrom theinternallink measurements.

2. BACKGROUND

2.1 NetworkAn IP network is madeupof IP routersandIP adjacenciesbetween

thoserouters,within a singleautonomoussystemor administrativedomain. It is naturalto think of the network asa setof nodesandlinks, associatedwith the routersand adjacencies,as illustratedinFigure2. We refer to nodesandlinks thatarewholly internalto thenetwork asbackbonenodesandlinks, andreferto theothersasedgenodesandlinks.

In addition,it is helpful for IP networksmanagedby InternetSer-vice Providers(ISPs)to further classify the edge. As shown in thefigure,in generalthenetwork will connectto otherautonomoussys-temsandcustomersvia edgelinks. We categorizetheedgelinks intoaccesslinks, connectingcustomers,andpeeringlinks, whichconnectother(non-customer)autonomoussystems.A significantfractionofthe traffic in an ISP is inter-domainandis exchangedbetweencus-

Peer A

Peer B

Peering Links

Access Links

Peers

Customers

IP Network Backbone

Figure2: IP network componentsand terminology

tomersandpeernetworks. Traffic to peernetworks is todaylargelyfocusedon dedicatedpeeringlinks, as illustratedin Figure2. Un-derthetypical routingpoliciesimplementedby largeISPs,very littletraffic will transit the backbonefrom one peernetwork to another.Transittraffic betweenpeersmayreflecta temporarystepin networkconsolidationfollowing anISPmergeror acquisition,but shouldnotoccurundernormaloperatingcircumstances.

In largeIP networks,distributedroutingprotocolsareusedto buildtheforwardingtableswithin eachrouter. It is possibleto predicttheresultsof thesedistributed computations,from datagatheredfromrouter configurationfiles. (The resultsprovided hereare basedonrouterconfigurationfiles downloadedoncedaily). In our investiga-tion, we employ a routing simulatorsuchasin [19] that makesuseof staticallyconfiguredBorderGateway Protocol(BGP)andInteriorGateway Protocol(IGP) topologyinformationgleanedfrom thecon-figurationfiles. In operationalIP networks, this informationis quitestableon thetime scalesof interest.

2.2 Traffic DataIn IP networks today, link load measurementsare readily avail-

ablevia theSimpleNetwork ManagementProtocol(SNMP).SNMPis uniquein that it is supportedby essentiallyevery device in an IPnetwork. The SNMP datathat is availableon a device is definedina abstractdatastructureknown asa ManagementInformationBase(MIB). An SNMPpoller periodicallyrequeststheappropriateSNMPMIB datafrom a router(or otherdevice). Sinceevery routermain-tainsacyclic counterof thenumberof bytestransmittedandreceivedon eachof its interfaces,we canobtainbasictraffic statisticsfor theentire network with little additionalinfrastructuresupport– all weneedis anSNMPpoller thatperiodicallyrecordsthesecounters.

Thepropertiesof datagatheredvia SNMPareimportantfor imple-mentationof a usefulalgorithm– SNMPdatahasmany limitations.Data may be lost in transit (SNMP usesunreliableUDP transport;copying to our researcharchive may introduceloss). Datamay beincorrect(throughpoor router vendorimplementations).The sam-pling interval is coarse(in our case5 minutes).Many of the typicalproblemsin SNMPdatamayberemovedwith minimal artifactsus-ing simple techniques. For instance,using hourly traffic averages(with fiveminutedatapolls) mitigatestheeffectof missingdatasub-stantially. Slightly moresophisticatedmethodsof anomalydetectionandinterpolationproduceevenbetterresults,but weshallusesimplehourly datafor thepurposesof this study, ashourly (or longer)dataarecommonlydealtwith by many ISPs(with fiveminuteor finerdatakeptfor brief periodsfor trouble-shootingandalarming).

We useflow level datain this paperfor validationpurposes.Thisdatais collectedat the routerwhich aggregatestraffic by IP sourceanddestinationaddress,andTCPport numbers.This level of granu-larity is sufficient to obtaina realtraffic matrix [13], andin thefuturesuchmeasurementmay provide direct traffic matrix measurements,but at presentlimitations in vendorimplementationsprevent collec-tion of this datafrom theentirenetwork.

2.3 TerminologyFor thepurposeof computingtraffic matrices,without lossof gen-

erality, weassumethatall accessandpeeringlinks terminateatEdgeRouters (ERs),andthat all remainingroutersareBackboneRouters(BRs) that only terminatebackbonelinks. (We can always insertdummyERsto force the assumptionto be true.) Figure3 providesa simplifiednetwork topologyto illustratetheterminologyWe makea furtherdistinctionthat links betweenBRsarecore links, andlinksbetweenERandBR arenon-corelinks.

BRBR

ER ER

PoP A

PoP B PoP C

core backbone link

non−core backbone link

edge link

2 3

41

Figure 3: A simplified network topology to illustrate the termi-nology usedhere. Edge Routers (ERs) are shown shaded,whileBackboneRouters (BRs) areunshaded.

Given two ERs HI� and HJ , the traffic betweentheseedgerouters�LK�2 is definedasthetotal amountof traffic that is entersthenetworkat HI� andexits at HJ , with MNK � � �LK�2 � theassociatedmatrix. Wemayalsodefinetraffic matricesbetweenBRs MNO in asimilarmanner,wheretheelementsrefer to traffic enteringandleaving thecore.Wewill often refer to a vectorform of the traffic matrix � in which theindicesof thevectorreferto source/destinationpairs.

Theremaybemorethanoneroutebetweentwo routersevenusingonly shortestpaths.We assumethattraffic will beevenly distributedacrossall suchroutes(thoughour methodcanbe easilyadaptedtohandleunevendistributions).

Onecouldcomputetraffic matriceswith differentlevelsof aggre-gation at the sourceand destinationendpoints,for instance,at thelevel of PoPto PoP, or routerto router, or link to link [20]. In thispa-per, we areprimarily interestedin computingrouterto routertrafficmatrices,which areappropriatefor a numberof network andtrafficengineeringapplications,andcanbe usedto constructmorehighlyaggregatedtraffic matrices(e.g. PoPto PoP)usingrouting informa-tion [20].

3. SOLUTIONIn this sectionwe provide our method,termedtomogravity, for

computingthe traffic matrix from link data. As its nameindicates,themethodconsistsof two basicsteps– agravitymodelingstep,anda tomographicestimationstep:

1. In thegravity modelingstep,an initial solutionis obtainedbysolving a gravity model using edgelink load data. We alsoincorporatestaticISProutingpolicy informationandexplicitlymodelthetraffic exchangedwith peernetworks.

2. In the tomographicestimationstep,the initial solution is re-finedby applyingquadraticprogrammingto find asolutionthatminimizesthedistanceto theinitial solution(in weightedleast-squaresense)subjectto the tomographicconstraints.We alsoapplyknowledgeof thenetwork routingandtopologyconfig-urationto significantlyreducetheproblemsize. Iterative Pro-

portionalFitting (IPF) is usedto ensurenon-negativity of theresults.

Below we discusseachstepin a separatesubsection,followed by abrief summaryof thecompletealgorithm.

3.1 Gravity ModelingOneof thesimplestapproachesto computinga traffic matrix is the

gravity model[2, 3, 4, 5, 6]. Gravity models,takingtheir namefromNewton’s law of gravitation, arecommonlyusedby socialscientiststo modelthemovementof people,goodsor informationbetweenge-ographicareas[5, 6]. In Newton’s law of gravitation theforceis pro-portionalto theproductof themassesof the two objectsdividedbythedistancesquared.Similarly, in gravity modelsfor cities,therela-tive strengthof the interactionbetweentwo citiesmight bemodeledasproportionalto theproductof thepopulations.A generalformula-tion of agravity modelis givenby thefollowing equation:

$%�2 � @N�P�� QR �S (4)

where$%�S is the matrix elementrepresentingthe force from � to�;@N� representstherepulsivefactorsthatareassociatedwith “leaving”

from � ; I representsthe attractive factorsthat areassociatedwith“going” to

�; and

R �S is a friction factorfrom � to�.

In ourcontext, wecannaturallyinterpret$%�S asthetraffic volumethatentersthenetwork at location� andexits at location

�, therepul-

sionfactor@N� asthetraffic volumeenteringthenetwork at location� ,andtheattractivity factor Q asthetraffic volumeexiting at location�. Thefriction matrix ( R �2.- encodesthelocality informationspecific

to differentsource-destinationpairs.Theinferenceof all @TFQ@ fric-tion factorsis anequivalentproblemof thesamesizeastheinferenceof the traffic matrix itself. Accordingly, it is necessaryto approxi-matetheactualfriction matrix usingmodelswith fewer parameters.In thispaper, weshallassumeacommonconstantfor thefriction fac-tors,whichis arguablythesimplestamongall possibleapproximationschemes.The resultinggravity modelsimply statesthat the trafficexchangedbetweenlocationsis proportionalto thevolumesenteringandexiting at thoselocations.Our resultsshow that,remarkably, thisgravity modelwhencombinedwith detailedknowledgeof ISProut-ing policies,is ableto matchtheactualInternetdatavery well. Onepossibleexplanationfor this is thatgeographiclocality is notamajorfactor in today’s Internet,ascomparedto ISP routing policies. Aslongasthegravity modelcapturestheessenceof theroutingpolicies,it becomesvery accurateandthechoiceof thefriction factorsis lesscritical.

Notethatwe do not expectour gravity modelto accuratelymodelthe traffic betweenall source-destinationpairs. In fact, onewouldnaturallyexpectcertainpairsof locationsto standout from theover-all distribution,simplydueto theirspecificcharacteristics(e.g.goingthroughtransoceaniclinks). A key insightof thetomogravity methodis thatweonly needthegravity modelto capturetheoverall distribu-tion; weexpectthetomographicestimationstepto correctmostof theviolationsin theassumptionsunderlyingthegravity modelandthussignificantlyimproving the overall accuracy. It is certainlypossibleto furtherimprovethemethodby usingmoreaccurategravity modelswith additionalparameters.However, asweshow laterin Section4.3,thetomogravity methodin its currentsimpleform is alreadyremark-ablyaccurate.Themargin for improvementmaybelimited.

Anotherimportantissueconcerningthegravity modelis the levelof aggregation. Intuitively, we needtheaggregationlevel to besuffi-ciently high sothat thetraffic exchangedbetweendifferentlocationsis notsensitive to thedetailedcompositionof thetraffic. Ontheotherhand,whentheaggregation level is too high (e.g. PoPto PoP),ISProuting policiesoperatingat a moregranularlevel may have a pro-found impactandcanintroduceserioussystematicdistortionto theoverall traffic pattern. In our tomogravity method,we applygravitymodelsat thelink to link level, which is thefinestlevel of resolution

obtainablewith SNMP data. We caneasilyuserouting informationto obtainmorehighly aggregatedtraffic matrices[20].

We formally presenttwo (link to link) gravity modelsbelow. Thefirst oneis quitesimpleandis primarily usedto provide insight intotheapproach.Thesecondapproachincorporatesmorerealisticrout-ing assumptions,andits performanceis therebymuchimprovedoverthesimplemodel.

3.1.1 A SimpleGravityModel

In this very simplegravity model,we aim to estimatetheamountof traffic betweenedgelinks. Denotethe edgelinks by U3V , U D , ....We estimatethe volume of traffic �I(GU+�+*WUW.- that entersthe networkat edgelink U,� and exits at edgelink UW . Let �IX YZ X YE[ ()U,�,- denotethetotal traffic thatentersthenetwork via edgelink U+� , and �Q\3]_^Z X YE[ ()U,�,- thecorrespondingquantityfor traffic thatexits thenetwork via edgelinkU+� . Thegravity modelcanthenbecomputedby eitherof

�I(GU+�,*,UW.- � � X YZ X Y_[ ()U+�W- �J\�]�^Z X Y_[ ()UW.-/ � \�]�^Z X YE[ ()U/ - *

�I(GU+�,*,UW.- � �IX YZ X Y_[ (GU+�,-/ � X YZ X Y_[ ()U/ - � \�]�^Z X YE[ ()UW.- <

Thefirst equationstatesthat the traffic matrix elements�Q()U+�,*,UW.- arethe productof the traffic enteringthe network via edgelink U+� andthe proportionof the total traffic leaving the network via edgelinkUW , while thesecondis reversedandis identicalundertraffic conser-vation– that is, theassumptionthat the interior network is neitherasource,nor sink of traffic. While this assumptionis violated (e.g.,protocolsrunningin the network interior act sourcesandsinks,andpacket dropsact assinks) the volumesinvolved are insignificantinthenetwork considered.Mostnotablytheactualresultsfrom thetwoequationsarealmostidentical.

3.1.2 GeneralizedGravityModelIt ispossibletogeneralizethesimplegravity modelof Section3.1.1

to take into accounta wide rangeof additionalinformationprovidedby link classificationandroutingpolicy. In this Section,we will in-corporatenew informationto modellargeISProutingpolicies.

Typically, in largeNorth-AmericanISPs,themajority of traffic isexchangedbetweennetwork customersandnetwork peers.Thepat-ternsandhandlingof customerandpeertraffic arequalitatively dif-ferent,andthis hasa largeimpacton thetraffic matrix. Furthermore,this peeringtraffic hasa largeimpacton every aspectof network de-sign andengineering,andso estimatingthe associatedtraffic matri-cesis very important. In this Section,we adaptthegravity modeltospecificallydifferentiatebetweencustomerandpeeringtraffic.

We assumethatthenetwork hasa setof peerslabeled VE*,` D * <5<5< ,andexchangestraffic with peera� over a setof edgelinks dedicatedto thispeer. Thisis commonlytermedprivatepeeringandis thedom-inantmodeof peeringfor large IP backbonestoday. We alsohave asetof customeraccesslinks, labeledb�VE*,b D * <5<5< , andasetof peerlinkslabeledc�V_*,c D * <5<5< . We denotethesetof edgelinks carryingtraffic topeer `a� by d�� , and the set of all peerlinks by d . We denotethesetof all customeraccesslinks by e . SNMPmeasurementsprovidevolumesof traffic on all edgelinks,

�: � X Y � \�]�^Z X YE[ ( � - , wherethe super-

scriptsin (out) denotestraffic into (out of) thebackbone.Thetrafficentering,or exiting thenetwork to peera� , is

�JfgihWh,j (G`a�,- � k=l.mon �JfZ X Y_[ ()c1-3*where& � in or out.

We will develop the equationsfor a gravity modelunderthe fol-lowing additionalassumptions,which reflectdominantInternetandISProutingpolicies:

Transit peer (peeringlink to peeringlink) traffic. We assumethatthevolumeof traffic thattransitsacrossthebackbonefromonepeernetwork to anotheris negligible.

Outbound (accesslink to peeringlink) traffic. We apply theproportionalityassumptionunderlyinggravity modelingon apeer-by-peerbasis:that is, thetraffic exiting to a specificpeercomesfrom eachaccesslink in proportionto thetraffic on thataccesslink. We assumethat all of the traffic from a singleaccesslink to the given peerexits the network on the samepeeringlink (determinedby the IGP andBGProutingconfig-uration). We denotethe exit peeringlink for traffic from ac-cesslink b�� to peer `p by $q()b��+*,`p.- . This may be derivedfrom routingconfigurationinformation(See[13, 15].) Theas-sumptionis typically truein practice,exceptfor examplewhenshort-termloadbalancingis performed.In suchsituations,ourmethodcouldbesupplementedwith availablestatisticson theaffectedprefixes, thoughour experienceis that the impact issmallanddoesnotaffecttheaccuracy of thetraffic matrixcom-putations.

Inbound (peeringlink to accesslink) traffic. A network opera-tor haslittle controlover theinjectionof traffic into its networkfrom peernetworks. Accordingly, we assumethat the trafficenteringfrom a given peeringlink is split amongsttheaccesslinks in proportionto their outboundtraffic.

Inter nal (accesslink to accesslink) traffic. We assumethatthe fractionof internaltraffic from a givenaccesslink bP� to asecondaccesslink b1 is proportionalto thetotal traffic enteringthenetwork at b�� , andcomputethetraffic betweenthelinks bynormalization.

Undertheseassumptionstheoutboundtraffic fromaccesslink bP��re to peeringlink c1s8rCdt is

� \�]�^�u.\�]EYEv ()b��,*,c1sw- �xzy {| y {�}:~�� n,��C�E����� x y {| y {�} ~�� � � �J\�]�^g.h,hWj ()`�.-�*

if c�s � $q()b��,*,`�.-3*� * otherwise<Theinboundtraffic from peeringlink c1� to accesslink b1 is

� X Y3u.\�]EYEv ()c��+*,b1.- � � X YZ X Y_[ ()c��,- �Q\3]_^Z X Y_[ ()b�.-� � l5��� \3]�^Z X Y_[ (Gb/ - <

Theinternaltraffic from accesslink bP� to accesslink b1 is

� X Y�^ h,j YE� Z ()b��+*,b�i- � �QX YZ X YE[ ()b��W-� � l.� � X YZ X Y_[ ()b/ - � \�]�^X Y3^ hWj Y5� Z ()b1.- <

where

� \3]_^X Y3^ h,j YE� Z ()b�.- � � \3]_^Z X YE[ ()b1.-o� k � l.m � X Y3u.\�]EYEv ()c/ *,b1.-

� � \3]_^Z X YE[ ()b1.- ��� k5n�l.m �QX YZ X YE[ ()c��,-� � l5��� \3]�^Z X Y_[ (Gb/ - <

Thecomplexity of thealgorithmis >?()��D5- in thenumberof edgelinksbeingconsidered,but thenumberof operationspertermissmall.Computationof thegeneralizedgravity modelfor thecompletenet-work in question(of theorderof 1000routers)took lessthan3 sec-onds(on a 336MHz Ultrasparc-IIprocessor).Despitethespeedandsimplicity of gravity modelssuchasthatexpressedabove,themodelshave the significantdrawbackof not guaranteeingconsistency withtheinternallink measurementsin thenetwork. Remedyingthis leadsusto thetomographicmethodsdiscussednext.

3.2 TomographyNetw� ork tomography, asmentionedearlier, is theproblemof deter-

mining theend-to-endtraffic matrix from link loads.Thelink trafficis the sumof the traffic matrix elementsthat areroutedacrossthatlink, andsowemayseeourproblemasfollows. Wehaveasetof ob-servables��� (G&�V_*,& D * <5<5< *,&���-

x, thetraffic (asmeasuredin packets

or bytes)thattraversesthe � links of thenetwork duringsomeperiod,whichderivefrom thetraffic matrix following equation(2) where� isthetraffic matrix, written asa columnvector �Q� ( 4 VE* 4 D * <5<5< * 4.� -

x,

wherethe � traffic matrix elements,4E�

, arethe traffic betweenthe� th source/destinationpair, and ��� � L� � � is the ��F�� routingmatrixwhere

L� � � � � � * if traffic for � traverseslink �� * otherwise (5)

where� � � is thefractionof traffic from source/destinationpair � thattraverseslink � . In someformulationsof the problem,an additiveterm � appearson theright handsideof equation(2) to modelmea-surementerror.

We needto solve the inverseproblemto obtain � . For generaltopologiesandroutingtherearetypically many moreunknownsthanconstraints,andsoequation(2) is highly under-constrainedanddoesnot have a uniquesolution. In order to infer � it is useful to adopta traffic modelof somekind – in the previous work on network to-mography the model allows the addition of higher-order statisticalconstraints.Ourapproachis not to incorporateadditionalconstraints,but ratherto usethegravity modelto obtainaninitial estimateof thesolution,whichneedsto berefinedto satisfytheconstraints.Thede-tailsarepresentedbelow, but first it is importantto reducethesizeoftheproblemto makecomputationof thesolutionmoremanageable.

3.2.1 ReducingtheComputationalComplexityOne of the main problemswith practicalapplicationof any to-

mographictechniqueis the sizeof the routing matrix. The chosendatasamplesin the studycover on the orderof a thousandrouters,which would leadto an intractablylarge problem(with millions ofunknowns).Oneapproachis to dealonly with smallerproblems.Forexample,[7] examinesPoPto PoPtraffic matrices.In this paper, wefocuson computingBR to BR traffic matrices,which aregenerallymuchmoreuseful for traffic engineeringtaskssuchasload balanc-ing, andareabsolutelynecessaryfor link/router failureanalysis.Wecandirectly derive PoPto PoPtraffic matricesfrom BR to BR ma-tricesusingrouting information. However, even if we only considerbackbonerouterstheremaybeof theorderof onehundred,leadingto a problemwith over ten thousandunknowns, which is ordersofmagnitudemorethantheavailableconstraintson link traffic volume.

In this sectionwe developtechniquesthatcansignificantlyreducethenumberof unknowns(in our caseby a factorof 10 for theBR toBR traffic matrix) andthusmakingthecomputationof traffic matrixbothmoreaccurateandmoreefficient.

Our key observation is that many BR to BR matrix elementsareemptyasa resultof the fact that theremay be multiple BRs withineachPoP, andso traffic will flow only betweenthe closestof these(asdeterminedby IGP routing). For instanceconsiderthesimplifiedtopologyshown in Figure3. Here thereare two BRs in eachPoP,connectingERswithin thePoPwith redundantlinks. Givenshortestpathrouting(andequallink weightson backbonelinks), onecanseethat all of the traffic from PoPB, to PoPC will traversethe routethroughBRs2 and3, while therewill beno traffic enteringtheback-bonenodesatBR 1 anddepartingatBR 4. While this is averysimpleexample,in operationalIP networks,thesetof pathsconsistentwithIP routingwill typically besignificantlylessthanthesetof all pathsbetweenrouterpairs.

ConsidertheBR to BR traffic matrix,anddenotethetraffic matrixfrom BR 0 � to 0 by �LO�S . Weusethefollowing simplealgorithmforremoving all theemptydemands:

1 Initially markall elementsof theBR to BR traffic matrixasempty2 For eachpairof ERssimulatetheIGPasin [19] to find theshortest

pathsbetweenthesourceanddestinationrouter;3 For eachpath,let 0 � and 0 beits first andlastBRsrespectively,

andmark �LO�2 asnotempty;

4 Remove all �LO�S that remainempty. This stepis equivalentto re-moving elementsfrom � thatwill bezerobecausethecorrespond-ing routeis notused(unlessfailuresoccur).

Wecanexploit thetopological equivalenceof ERsto avoid havingto run IGP simulationsfor all possiblepairs of routers,which canbe prohibitive due to their large number. We considertwo ERs tobe topologically equivalentif they connectto thesame(non-empty)setof BRs andthe IGP weightson the correspondinglinks are thesame1. Wegroupsuchequivalentedgerouterstogether, andconsiderthemasa singleEdge RouterEquivalenceClass(EREC).The keyobservationis thattheroutesbetweenthecomponentERsof thesamepair of ERECswill be the sameexcept for the first and last links.Consequently, we only needto run oneIGP simulationfor eachpairof ERECs.We foundthatcomputingrouteson this basisreducesthecomputationalburdenby at leasta factorof 20.

After eliminating all the empty demandswe are able to reducethenumberof unknownsby a factorof 10, therebyturninga highlyunder-constrainedproblemintoamoderatelyunder-constrainedprob-lem,while makingthecomputationordersof magnitudefaster.

The routing matrix is requiredfor all methodsexcept the simplegravity model.Thetime takenin computingtheroutingmatrix dom-inatesall otheralgorithmsweuse,takingtwo to threeminutes.How-ever, notethat this costcanoften be amortizedover multiple trafficmatrix computationsbecausewe only needto recomputetheroutingmatrix whenthenetwork topologychanges.Thealgorithmto reducetheproblemsizecanbeperformedaspartof computingthe routingmatrix, with a computationalcostthat is a very small marginal coston topof computingtheroutingmatrix itself.

3.2.2 Quadratic ProgrammingFromthe(link to link) gravity model,we caneasilyobtainanes-

timateof the BR to BR traffic matrix ( M O ) usingrouting informa-tion [20]. The link to link traffic matrix elementsareroutedacrossthebackbone,andwethencomputethetraffic volumesbetweeneachpair of backbonerouters(notethatwe loosethedistinctionbetweenpeeringandaccesstraffic in this step). We term the resultingesti-mateasthe gravity modelsolution (denotedas �.  ). We thenrefinethegravity modelsolutionby usinga least-squaresolutionthatmin-imizesthe Euclideandistanceto the gravity modelsolutionsubjectto the tomographicconstraints.More specifically, we would like tosolve thefollowing quadraticprogram

¡B¢�£ ¤ � � �. w¤¥ < ¦§< ¤ � 4 ��& ¤¨¢ ¥ ¡B¢�£©¢S¡B¢�ª5«5¬where ¤J<©¤ is the � D normof avector(i.e., theEuclideandistancetotheorigin).

We alsoinvestigateweightedleast-squaressolutionsin which theprojectiononto the subspaceis not orthogonal,but ratherweightedby a functionof thesizeof theestimatedtraffic matrixelements( �.  ).That is, we use ¤ ( � � �i  -3­i® ¤ asthe objective function to mini-mize in theabove quadraticprogram,where ® is theweightvector,and ­ is theelement-by-elementvectordivision. Weinvestigatethreeweightingschemesin thispaper;theweightfor eachtermof thetraf-fic matrix is either:constant,linearly proportionalto thetermsin thegravity modeltraffic matrix,or proportionalto thesquareroot of thegravity model.V Theremay be a multiple layer hierarchy of ERswithin a PoP, andlowerlayerERspasstraffic to BRsonly throughthehigherlayerERs.Wemayremove thelower layerof ERsfrom considerationasweseetheir (network impacting)traffic at thehigherlayer.

Figure 4 illustratesthe approachin a simple casewith two un-knowns¯ (unknown traffic alongtwo routes),andoneconstraint(oneknown link load). The constraint(s)specifya sub-spaceof the pa-rameterspace(in this casejust a line), while the gravity model isone particularsolution. The simple least-squaresapproachis justanorthogonalprojectionof thegravity modelsolutiononto thecon-straintsub-space.Theweightedleast-squaressolutiongivesdifferentweightsto differentunknownsin thesolution.

gravity

constraintsubspace

model

least−squares solution

weighted least squares

Figure 4: An illustration of the least-square solution. The pointshows the gravity model solution, and the dashedline shows thesubspacespecified by a single constraint equation. The least-square solution is simply the point which satisfiesthe equationwhich is closestto the gravity modelsolution. The weightedleast-squaressolution givesdiffer ent weight to differ ent unknowns.

Notethatthetomographicconstraintsmaybeill-poseddueto pos-sibledependency amongdifferentlink constraints.Furthermore,theconstraintsmay not be satisfiabledueto error andnoisein the linkload dataor possiblerouting changesthat are not capturedby thetopology data. The standardtechniquefor dealing with ill-posedquadraticprogramsis to useSingular-ValueDecomposition(SVD)of theroutingmatrix � to computeits pseudo-inverse.Theresultingsolution is the closestto the initial solution �i  amongall solutionsthat minimize the discrepancy against the tomographicconstraints( ¤Q� 4 �¨& ¤ ). Routinesto computethepseudo-inverseareavailablein many numericalcomputingpackages(for instancesee[21]). Wehave implementedour methodin Matlab,andtheactualalgorithmissosimplethatit only takes6 linesof code(seetheAppendix).

Theworst casecomplexity of theabove algorithmis linear in thenumberof unknowns (elementsof the traffic matrix), andquadraticin thenumberof constraints,however, in practicethecomplexity ofsingularvaluedecompositionmethodsis generallylessthanthis. ForinstancetheSVD usedin LAPACK [21], andthenceMatlab (whichusestheseroutines)areusuallybetterthanthis complexity estimatewould indicate.In reality, thealgorithmis very fast,with thecompu-tation timeson a 336 MHz Ultrasparc-IIprocessor(runningSolaris7) all significantlyunder2 seconds.

Oneadditionallocusof complexity is that the least-squarealgo-rithm mayresultin negativevalues,whicharewithoutphysicalmean-ing. Onecanavoid this by viewing theproblemasa constrainedop-timizationproblem. However, a simpleiterative procedureprovidesa fastandeffective alternative. Specifically, we useIterative Propor-tionalFitting (IPF) assuggestedin [10] to ensurenon-negativity. Wedo not modelhigherorderstatisticsof theprocess,andtherefore,theinitial conditionwe useis not ascomplex asthatin [10]. For theini-tial estimate,we simply usethe traffic matrix estimatedabove, withzeroreplacingthenegativeelementsof thematrix. IPFthenproceedsby successively refiningtheestimateusingthesamemethodas[10].In practiceit only takesa few iterationsto reduceerrorsin thecon-straintequationsto thepointatwhich they arenegligible.

3.3 The CompleteAlgorithmBelow wesummarizethecompletetomogravity method.1 Apply thegeneralizedgravity modelto obtaina link to link traffic

matrix.2 Transformtheabove traffic matrix into a BR to BR traffic matrix,

anduseit asour initial gravity modelsolution( �.  ).3 Reducethe complexity of the tomography problemby removing

emptyBR to BR demands.Topologicalequivalenceof ERscanbeexploitedto speedup theroutecomputationby a factorof 20.

4 Apply singularvalue decompositionto solve the quadraticpro-gram and find a solution that minimizes its distanceto �.  (inweightedleast-squaresense)subjectto thetomographicconstraints.

5 Replacenegative valueswith zero,andperformIPF to obtainanon-negativesolutionthatsatisfiestheconstraints.

4. VALID ATIONIn this section,we validateour methodsusingactualInternetdata.

We first describethebasicvalidationmethodology. We thenvalidatethe underlyingassumptionsfor the gravity modelswe use. This isfollowed by a detailedcomparisonof the quality of the resultsob-tainedby differentmethods. Finally, we presentsomepreliminaryresultson therobustnessof thetomogravity method.

4.1 MethodologyIn anidealenvironment,wewould like to validatetheresultsusing

a consistentset of real traffic matrix, topology, and link measure-ments.Unfortunately, we do not have thecompletesetof flow leveldataacrossthe edgeof the network (dueto problemsin the vendorimplementationsof flow collection). Furthermore,the routing datawehaveavailablecomesin 24hoursnapshots,andtheflow level datais sampleddata,with its own limitations.Therefore,it is notpossiblefor usto obtaina completetraffic androutingmatrix consistentwiththeactualSNMPlink measurements.

Wesolvetheproblemof providing aconsistentsetof traffic, topol-ogy andlink measurementdataasfollows. We first derive a setofhourly (partial) traffic matricesbasedon direct flow level measure-mentsusing the methodologydescribedin [13]. We thensimulateOSPFroutingusingthetopologyandroutinginformation.Fromthiswe may computea routing matrix � , and thenderive a setof linkmeasurements� from equation(2). Thus the traffic matrix � , theroutingmatrix � andthemeasuredlink loads� areall consistent.

We can then estimate °� for a problem for which we know the“ground truth” – the real traffic matrix. This approachhasthe ad-ditional advantagethat it isolatesthe impactsof measurementerrorsandinconsistenciesfrom theperformanceof thealgorithms(thetopicof this paper).We cantherebystudythe impactof measurementer-rorsseparately, andmoreprecisely, aswecaninducetheerrors,ratherthaninfering themfrom multiple (possiblyinconsistent)datasets.

Notethatweuseusederivedlink loadmeasurementsfor validationpurposesonly. In practice,ourmethodis applicableto directlink datasuchasonewould obtainfrom SNMP, andwe have in fact appliedthemdirectlyto SNMPdatafor operationaltasksin capacityplanningandnetwork management.

4.2 Gravity ModelsIn this section,we validatethe basicassumptionsunderlyingour

gravity models,in particulartheassumptionon constantfriction fac-tors,which is thesimplestpossibleapproximationto the @±FL@ fric-tion matrix.

By simple rearrangementsof the basicequationsof the gravitymodel,it is clearthatwhentheestimatedfriction factorsareconstant,theactualfriction factorsareconstantif andonly if theratiobetweentheestimatedandtheactualmatrixelementsareconstant.Clearlytheconstantshouldbe1 asthegravity modelsmaintainthecorrecttotaltraffic volumethroughappropriatenormalization.As a result,to test

if theactualfriction factorsareconstant,oneonly needto verify thatthepoints² ()$ ��2 *,$´³�2 - lie onastraightline passingthroughtheorigin.It is easyto show that the sameanalysisappliesfor the simpleandthegeneralizedgravity modelsintroducedin Section3.1.

real matrix elements

estim

ated

mat

rix e

lem

ents

(a)Simplegravity model

real matrix elements

estim

ated

mat

rix e

lem

ents

(b) Generalizedgravity model

Figure 5: A comparisonof the real traffic matrix elementsto thetraffic matrix elementsestimatedby the gravity model. The soliddiagonal line showsequality, while the dashedlinesshow ! 20%.

Figure5 illustratesthetwo link to link gravity modelsfor onehourchosenrandomly from June2002. In eachcasewe plot the truevalueof theelementsof thetraffic matrix versustheestimatedvalue.Clearly, the closerthe pointsclusteraroundthe diagonal,the moreaccuratethemodel. It appearsthat the generalizedgravity modelismoreaccuratethanthesimplemodel.

To quantify the linear associationbetweenthe real and the esti-matedmatrix elements,we computethe correlationcoefficient foreachhourly traffic matrix,alongwith theleast-squareregressionline.We needto verify two facts: � ) thecorrelationcoefficient is closeto1 (sothat thepointsarecloselycenteredaroundtheregressionline),and��� ) theinterceptof theregressionline is closeto 0.

Figure6 shows thedistribution of thecorrelationcoefficientsovertheentireJune2002dataset.For thesimplegravity model,thecorre-lation coefficient typically lies between0.65and0.8,which aresig-nificant,but notaslargeaswewould like. In contrast,thecorrelationcoefficientsfor thegeneralizedgravity modelareconsistentlyabove0.9,indicatingverystronglinearassociation.

We next computethe95%confidenceinterval for the interceptoftheregressionline undertheassumptionthattheregressionresidualsis Gaussian.We find that the 95% confidenceinterval covers0 on

0 0.2 0.4 0.6 0.8 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

correlation coefficient

% o

f hou

rly tr

affic

mat

rices

simple gravitygeneralized gravity

Figure6: Distrib ution of the correlation coefficientsbetweenrealand gravity estimateof traffic matrix.

all 507hourly traffic matricesfor boththesimpleandthegeneralizedgravity model.This indicatesthattheinterceptis indeedcloseto 0.

Theseresultsshow that the generalizedgravity model is signifi-cantlymoreaccuratethanthesimplegravity model,whichhighlightsthe importanceof explicitly taking into accountISProutingpoliciesin gravity modeling. Moreover, despiteits simplistic assumptionsaboutthetraffic (i.e. proportionality, andconstantfriction factor),themodelcancapturetheoveralldistributionof thematrixelementsverywell. As a result,the resultingestimatecanprovide a goodstartingpoint for thetomographicestimationstep.

4.3 PerformanceThe main methodof validationwe useis to directly comparethe

estimatedtraffic matrixelementswith thetruetraffic matrix. Figure7providesa comparisonof methodsfor computingthegravity matrixfor onehourchosenrandomlyfrom June2002. In eachcasewe plotthe true value of the elementsof the traffic matrix versusthe esti-matedvalue. (Theexactvaluesof traffic on thenetwork in questionareconsideredproprietaryso the scalesarenot present,but arenotneededfor thecomparisonto bevalid.) Thesoliddiagonalline showsequalityandthedasheddiagonallinesshow ! 20%. For a base-linecomparisonwe first show in Figure7 (a) the least-squaresolutiontoequation(2), to demonstratethatadirectsolutionof theequationdoesnot producea goodresult. Figure7 (b) shows the resultof thesim-ple gravity model, which are also fairly inaccurateoverall, thoughbetterthanthe pureleastsquaresapproach.Figure7 (c) shows thegeneralizedgravity model,andwe canseea dramaticimprovementover the previous algorithms,thoughit is still not perfect. Finally,Figure 7 (d) shows the resultswith the generalizedgravity model,andtheleast-squaressolution(usingsquareroot weights).This finalsolutionappearsremarkablygood.

To quantifythequality of theresults,it might seemidealto estab-lish a direct link to traffic andnetwork engineeringapplications,forinstancefor capacityplanning. However, theseinvolve complex al-gorithmsin their own right, andsowe shallattemptto presentsomesimplemetricsfor the quality of the resultswhich have somelink-ageto theengineeringrequirements.Therequirements(drawn fromextensive conversationwith engineersdoing capacityplanningandnetwork design)canbe approximatelystatedthus: we requirerea-sonablerelativeaccuracy for thelargerelementsof thetraffic matrix,andnot too largeabsoluteerrorsfor all matrix elements.How largeis a reasonableerror?In light of theothersourcesof errorin Internetplanning,andaccuracy measuredin a few ten’s of percentis quitetolerable,especiallyif theerrorsfor thelargerflowsaresmaller. Thisis furtherbackedup by work suchasthat in [22], which showedthat

actual matrix element

estim

ated

mat

rix e

lem

ent

(a)Raw leastsquaresactual matrix element

estim

ated

mat

rix e

lem

ent

(b) Simplegravity model

actual matrix element

estim

ated

mat

rix e

lem

ent

(c) Generalizedgravity modelactual matrix element

estim

ated

mat

rix e

lem

ent

(d) Generalizedgravity modelwith squareroot weightedleastsquares

Figure 7: A comparisonof the real traffic matrix elementsto the estimatedtraffic matrix elementsfor four differ ent algorithms. Allplots areshown on the samescale.The solid diagonal line showsequality, while the dashedlinesshow ! 20%.

Algorithm traffic matrixerrors corelink errorsinitial solution LSE weights RMSE(Mbps) RMSRE RMSE(Mbps) RMSRERaw yes N/A 89 65% 174 37%SimpleGravity no N/A 85 62% 260 54%SimpleGravity yes const 27 22% 9 2%SimpleGravity yes linear 28 24% 4 1%SimpleGravity yes root 25 21% 4 1%GeneralGravity no N/A 40 31% 117 24%GeneralGravity yes const 16 13% 4 1%GeneralGravity yes linear 16 13% 4 1%GeneralGravity yes root 15 12% 4 1%

Table 1: Performanceof the various algorithms over 507 individual hourly data setsfr om June 2002. The RMSREs are computedon the largestmatrix elementscomprising 75% of the network traffic, and the RMSE is computedon all the traffic matrix elementsfor the month.

at leastonenetworkingtask(routingoptimization)couldstill providesubstantialµ capacityimprovementsevenwhentheerrorsin the inputtraffic matricesweresignificantlylargerthanthosereportedhere.

It seemsthatasinglemetricis unlikely to cover thisadequately, soweshallpresentseveral.Two shown herearetheRootMeanSquaredError(RMSE),andtheRootMeanSquaredRelativeError(RMSRE),definedbelow.

RMSE � ��

¶�G·'V ( °&'�P�¨&'�,- D

< (6)

RMSRE()�Q- � �� x

¶�)·'V¸5n�¹ x

°&'�P��&��&'�D < (7)

TheRMSEgivesanoverallmetricfor theerrorsin theestimates,andtheRMSREgivesa relative measure,but notethattherelative errorsfor small matrix elementsarenot really importantfor engineering,andsowewhencomputingtheRMSREwetakeonly matrixelementsgreaterthan somethreshold� (and thereforewhen computingthemeannormalizedby the numberof matrix elementsgreaterthan � ,namely� x � ¶�)·'V � ¸ n ¹ x � ).

We computethesemetricsnot just to comparethe errors in thetraffic matricesthemselves,but alsoto comparetheobservables.Thatis weusetheestimatedtraffic matrix to predictthelink loadsoncorelinks, andcomparewith therealloadon theselinks.

Table1 summarizestheresultsfor thedifferentpossiblealgorithms– in theseresults � is chosenso that 75% of the traffic is captureby the RMSRE.Note that this truncationremoves elementsof thetraffic matrix whosemagnitudeis at most5% of thecapacityof thesmallestlink the the backbonenetwork being considered. Hence,evenquitelargerelativeerrorscanbetoleratedin thesetermsaslongasthe errorsarenot highly correlated(which shouldbe guaranteedby satisfyingthetomographicconstraints).

Thefirst threecolumnsof thetablespecifythealgorithmby its ini-tial conditions,whethertheLSE methodis applied,andif it is, whatweightsareused. The quality metricsabove arepresentedboth fortheerrorsin theestimatedtraffic matrix, andtheestimatedcorelinktraffic, andarebasedon our entiresetof datafor June2002(thoughtherearemissingdaysin this datasetthedatasetstill containsover500onehourintervalsof data).

TheRMSEresultsarequotedin Mbpsandthesemayall seemrela-tively smallin comparisonto thescaleof thenetwork – thebackboneconsistsprimarily of reasonablywell utilizedOC48andOC192links(2.5 and10 Gbpsrespectively), but in particularnote their relativevaluesfor comparison.The link errorsarenot as interestingasthetraffic matrixerrorsin somerespects,becausetherearemany possible(incorrect)solutionswhichcanproducethecorrectcounts.However,it is importantfrom a practicalstandpointto get theseat leastap-proximatelycorrect,andtheresultsshow thatwithouta tomographiccomponent,thelink traffic estimatesmaybequiteinaccurate.Whenatomographiccomponentis usedtheerrorsin thelink loadsareof thesamemagnitudeasthetargetederrorlevel usedin theIPFcomponentof thealgorithm,asrequired.

The resultsuniformly show2 that (a) the simplegravity model isbetter than the raw least-squaresalgorithm, and that (b) using theleast-squareswith agravity modelinitial conditionprovidesadefiniteimprovement,not just in theestimatedlink rates(which is what themethoddirectly improves), but also in the traffic matrix estimates.Furtherthegeneralizedgravity modelis betterthanthesimplegravitymodel. However the bestresultcomesfrom tomogravity, usingthegeneralizedgravity model with the weightedleast-squaresmethodusingsquareroot weights(thoughtheimprovementover usingother

D Otherperformancemetricsinvestigateduniformly supportthecon-clusiondrawn from Table1.

weightingsis small).Thestatisticsaboveappearto favor usingthesquarerootweighting

schemepresentedabove. We investigatefurther to seewhetherthisis really thecase.In Figure8 we vary � in computingtheRMSREandshow theRMSREfor thethreeweightingsconsidered,comparedwith thevolumeof traffic in thematrixelementsabove thethreshold.Theresultsshow thatnooneschemeis aclearwinner, but thesquareroot schemeseemsto win overall. We have investigatedsomeotherweightingschemesbut havenot foundasuperiorone.

0 20 40 60 80 1000

10

20

30

40

50

60

percentage of traffic

RM

SR

E fo

r to

p %

of t

raffi

c

rootlinearconst

Figure 8: The RMSRE for varying thr esholdfor the June 2002data. The x-axis shows the proportion of the network trafficfalling above the thr eshold, and the y-axis shows the RMSREfor the varying thr eshold. The resultsshow that the RMSRE issmaller for the larger matrix elements.

The plot alsoprovidesanotherview into the performanceof thealgorithm,asthethresholdusedis varied.For instance,Figure8 alsodemonstratesthattheresultsarebetterfor thelargermatrix elements(asthethresholdincreasesthevolumeof thethresholdedmatrix ele-mentsdecreases,andtheperformanceimproves).TheRMSREactu-ally dropsbelow 5%for thetop20%of traffic. Thisis averydesirableresult,but we canalsoseethat a very large proportionof the traffic(over90%)hasaRMSRElessthan20%.

Figure1 shows thedistribution of errorsfor thegeneralizedgrav-ity modelwith square-rootweightedleastsquares,andthe particu-larly noteworthy point to take away from this plot is that morethan30% of the estimatedmatrix elementshave a negligible error (lessthan0.5%). Theplot alsoshows the5th and95thpercentilesof thedistribution that lie within ! 23%. Furthermore,the plot shows thatthe resultsarenot biasedin onedirectionor the other. Thus,whenwe aggregatetheseresultsto higherlevels, for instancePoPto PoPtraffic matrices,theerrorsshouldnot addto make thingsworse.Weexaminewhetherthis is thecasein thefollowing figure.

Figure9 shows thepercentageof traffic (over thewholedataset)for which themagnitudeof the relative errorbelow somevalue(forthegeneralizedgravity modelwith square-rootweightedleastsquares).Onceagainweseegoodresults:morethan30%of thetraffic hasneg-ligible relative error, andnearly70%hasanerror lessthan10%forthe BR to BR traffic matrix. Further, aswe have notedearlier, it ispossibleto derivePoPto PoPtraffic matricesdirectly from theBR toBR traffic matrices.The dashedline in Figure9 shows the equiva-lentcurvefor suchmatrices,showing thattheperformanceis actuallyslightly betterover themajorityof thedistribution for thePoPto PoPmatrices.In fact, theRMSE for thePoPto PoPmatrix is 10 Mbps,while theRMSREis 11%.

Another methodto assessthe quality of the resultsis to look atthepredictionour estimatedtraffic matrix would make of propertiesof the network. A simpleexampleis presentedin Figure10, whichshows thepercentageof traffic in thelargest� matrixelements.The

0 10 20 30 40 50 60 70 80 90 1000

10

20

30

40

50

60

70

80

90

100

magnitude of the relative error (%)

% o

f tra

ffic

BR−BRPoP−PoP

Figure 9: The percentageof traffic where the magnitude of therelative error below somevalue (x-axis) for the June 2002data.The solid line shows the curve for the backbonerouter to back-bonerouter matrix, and the dashedline shows the curve for thePoP to PoP traffic matrix. Note that for nearly 30% of the trafficthe error is negligible.

plot shows how skewedthetraffic matrix is, andhow importantit isto estimatethe largestmatrix elementscorrectly. The figure showstheactualdistribution, andtheestimateddistribution – they arehardto distinguishbecausethereis almostnodiscernibledifference.

0 50 100 150 200 250 3000

10

20

30

40

50

60

70

80

90

100

largest N matrix elements

perc

enta

ge o

f tra

ffic

actualestimated

Figure 10: The percentageof traffic in the largest � matrix ele-ments for the real traffic matrix (solid) and the estimatedtrafficmatrix (dashed).

Finally, considerwhathappensto theestimatesovertime. Wewantto beableto answerquestionsaboutthevariability of thetraffic, or todetecttrendsor changesin thetraffic matrix. It is thereforedesirablefor theresultsto remainstable– thatis, for theerrorsin theestimatedtraffic matrix to remainapproximatelyconstantover time,evenasthetraffic matrix itself changes.Figure 11 shows two comparisonsofelementsof the traffic matrix over time, with their predictedvalues(notethegapsarewhereourarchive is missingdata).Thefirst showsa casewheretheerror is very small. Thetwo curves(theactualandestimatedmatrix elements)arealmostindiscernible,eventhoughwecanseecleardiurnal variationin the traffic, anda profounddrop inthetraffic at onepoint wherethenetwork topologychangeddramat-ically. The estimatestrack the traffic matrix perfectly. The secondplot showstheresultsfor acasethatis notsoaccurate,but onceagainwe seethat theestimatedvaluefollows thevariationof thetrueflowsmoothly. Thusevenwherethereareerrorsin the traffic matrix, theresultscanstill be usedfor a rangeof operationaltaskssuchasde-tectingtraffic changes.

06/03 06/08 06/13 06/18 06/23 06/28 07/03date

traf

fic m

atrix

ele

men

t

actualestimated

06/03 06/08 06/13 06/18 06/23 06/28 07/03date

traf

fic m

atrix

ele

men

t

actualestimated

Figure 11: Comparison for two matrix elementsover time. Inthe first caseweseethat the estimatedvalue tracks the true valuealmostexactly, while in the secondcasealthough there is an errorin the estimate,the error changesonly slowly over time. Gapsinthe data indicate missingdata in our archive.

4.4 RobustnessIn this section,we evaluatethe impactof measurementerrorsand

inconsistencieson theperformanceof thetomogravity methodby in-ducinganerrorterm � to thetomography constraints:

���º�B��» � < (8)

All thenumericalresultspresentedherearefor thetomogravity methodusingthegeneralizedgravity modelwith theweightedleast-squaresmethodusing squareroot weights. The error term � is formed bymultiplying the � with awhitenoiseterm.Specifically,

� ���C¼ �q( � *,½�-3* (9)

where ¼ is element-by-elementvectormultiplication,and �q( � *,½�- isa vectorwith randomentriesdrawn from a normaldistribution withmean0 and standarddeviation ½ . To ensurethe non-negativity of� ��� , wecapthenormaldistribution at1.

traffic matrixerrors corelink errorsNoiseLevel RMSE RMSRE RMSE RMSREnoisefree 15Mbps 12% 4 Mbps 1%½ � � < � � 15Mbps 13% 6 Mbps 1%½ � � < � " 16Mbps 14% 10Mbps 2%½ � � < ��¾ 19Mbps 17% 18Mbps 4%

Table2: Performanceof the tomogravity methodunder differ entnoise levels (0%, 1%, 2% and 4%) over 507 individual hourlydata setsfr om June2002.

Table2 summarizesthe performanceof the tomogravity methodunderdifferent levels of noise(0%, 1%, 2% and4%)3. The resultsshow thatwhile theinducednoisemakestheestimatedtraffic matrixlessaccurate(asonewould expect),the overall performancedegra-dationis small– theadditionalestimationerrorson bothmatrix ele-mentsandlink observablesareof thesamesize,or smallerthantheintroducederrorsontheobservables.Similarbehavior is alsoevidentin Figure12, wherewe vary the threshold� in computingtheRM-SREandshow theRMSREcomparedwith thevolumeof traffic in thematrix elementsabove thethreshold.While a moredetailedstudyofrobustnessmustthoroughlyinvestigatethetypesof measurementer-rorsseenin practice,theseresultsshow thatthetomogravity methodis robustto measurementerrorson theobservables.¿ � is chosento capturethetop75%traffic, asin Table1.

0 20 40 60 80 1000

10

20

30

40

50

60

percentage of traffic

RM

SR

E fo

r to

p %

of t

raffi

c

noise freesigma=0.01sigma=0.02sigma=0.04

Figure 12: The RMSRE for varying thr esholdfor the June 2002data. The x-axisshows the proportion of the network traffic cap-tur ed by the matrix elementsthat fall above the thr eshold,andthe y-axis shows the RMSRE for the varying thr eshold. The re-sults show the overall performance degradation due to inducednoiseis small.

5. CONCLUSIONWe have presenteda simple, fastandpracticalmethodcalledto-

mogravity for computingrouterto routertraffic matricesin large IPnetworks,basedon widely availablelink measurements.The tomo-gravity methodcombinesthebestfeaturesof gravity modelsandto-mographictechniques.Themethodscalesto handlenetworks largerthan any previously attemptedusing alternative methods,and it isvery fast (taking lessthan5 secondsfor a large network). Further-more,thisrouterto routertraffic matrixmaybeusedto directlyderiveanevenmoreaccuratePoPto PoPtraffic matrix.

We have validatedthemethodon thea largecommercialIP back-bonenetwork, wheredirectmeasurementsof the traffic matrix wereavailable for June2002. The resultsshow remarkableaccuracy forthe larger elementsin the traffic matrix: the estimatestypically fallwithin a few percentof true values. Theselarger valuesdominatenetwork andtraffic engineeringapplications.However, theaccuracyover all the matrix elementsis still reasonable,andTomogravity isnow beingusedfor a variety of capacityplanningandmanagementtasksin this network.

Thereareseveralavenuesfor further improvements.In particular,we would like to be ableto includeadditionalconstraintsin the so-lution. For instance,if onehaspartial flow level dataavailable,thiscouldprovide additionalconstraints.Anotheravenueof explorationis theuseof alternativestartingpointssuchasthatproposedin [7].

AcknowledgmentsWe would like to thankCarstenLund, FredTrue,JoelGottlieb,andTim Griffin, whosework in collecting,andmanagingthe datausedhere,madethis researchpossible,QuynhNguyenfor motivation,en-couragementandoperationalclue,andJenniferRexford,AmanShaikandChuckKalmanekfor many helpful comments.

6. REFERENCES[1] M. GrossglauserandJ.Rexford, “Passive traffic measurementfor IP

operations,” in TheInternetasa Large-ScaleComplex System(K.ParkandW. Willinger, eds.),OxfordUniversityPress,2002.

[2] M. Roughan,A. Greenberg, C. Kalmanek,M. Rumsewicz, J.Yates,andY. Zhang,“Experiencein measuringbackbonetraffic variability:Models,metrics,measurementsandmeaning(extendedabstract),” inACM SIGCOMMInternetMeasurementWorkshop, 2002.

[3] D. Lam,D. Cox,andJ.Widom,“Teletraffic modelingfor personalcommunicationsservices,” IEEECommunicationsMagazine:SpecialIssuesonTeletraffic ModelingEngineeringandManagementinWirelessandBroadbandNetworks, 35,pp.79–87,February1997.

[4] J.Kowalski andB. Warfield,“Modeling traffic demandbetweennodesin a telecommunicationsnetwork,” in ATNAC’95, 1995.

[5] J.Tinbergen,“Shapingtheworld economy:Suggestionsfor aninternationaleconomicpolicy.” TheTwentiethCenturyFund,1962.

[6] P. Pyhnen,“A tentativemodelfor thevolumeof tradebetweencountries,” WeltwirtschaftlichesArchive, 90,pp.93–100,1963.

[7] A. Medina,N. Taft, K. Salamatian,S.Bhattacharyya,andC. Diot,“Traffic matrixestimation:Existingtechniquesandnew directions,” inACM SIGCOMM, (Pittsburg, USA), August2002.

[8] Y. Vardi, “Network tomography: estimatingsource-destinationtrafficintensitiesfrom link data,” J. Am.Statist.Assoc.,91,pp.365–377,1996.

[9] C. TebaldiandM. West,“Bayesianinferenceonnetwork traffic usinglink countdata,” J. Amer. Statist.Assoc, 93,pp.557–576,1998.

[10] J.Cao,D. Davis, S.V. Wiel, andB. Yu, “Time-varyingnetworktomography,” J. Amer. Statist.Assoc, 95,pp.1063–1075,2000.

[11] A. Adams,T. Bu, R. Caceres,N. Duffield, T. Friedman,J.Horowitz,F. L. Presti,S.Moon,V. Paxson,andD. Towsley, “The useofend-to-endmulticastmeasurementsfor characterizinginternalnetworkbehavior,” IEEECommunicationsMagazine, May 2000.

[12] M. Coates,A. Hero,R. Nowak,andB. Yu, “Internettomography,”IEEESignalProcessingMagazine, May 2002.

[13] A. Feldmann,A. Greenberg, C. Lund,N. Reingold,J.Rexford, andF. True,“Deriving traffic demandsfor operationalIP networks:Methodologyandexperience,” IEEE/ACM TransactionsonNetworking, pp.265–279,June2001.

[14] W. FangandL. Peterson,“Inter-AS traffic patternsandtheirimplications,” in Proceedingsof IEEEGLOBECOM’99, (Rio deJaneiro,Brazil), pp.1859–1868,1999.

[15] N. Feamster, J.Borkenhagen,andJ.Rexford, “Controlling theimpactof BGPpolicy changeson IP traffic,” Tech.Rep.011106-02-TM,AT&T Labs– Research,Nov 2001.

[16] A. Dempster, N. Laird, andD. Rubin,“Maximum likelihoodfromincompletedatavia theemalgorithm(with discussion),” J. Roy. Statist.Soc.Ser., 39,pp.1–38,1977.

[17] J.Cao,S.V. Wiel, B. Yu, andZ. Zhu,“A scalablemethodforestimatingnetwork traffic matricesfrom link counts.” preprint.

[18] J.G. Klincewicz. privatecommunications,2000.[19] A. Feldmann,A. Greenberg, C. Lund,N. Reingold,andJ.Rexford,

“Netscope:Traffic engineeringfor IP networks,” IEEENetworkMagazine, pp.11–19,March/April 2000.

[20] A. Medina,C. Fraleigh,N. Taft, S.Bhattacharyya,andC. Diot, “Ataxonomyof IP traffic matrices,” in SPIEITCOM: ScalabilityandTraffic Control in IP NetworksII , (Boston,USA), August2002.

[21] E. Anderson,Z. Bai, C. Bischof,S.Blackford,J.Demmel,J.Dongarra,J.D. Croz,A. Greenbaum,S.Hammarling,A. McKenney, andD. Sorense,LAPACK Users’ Guide. SIAM, 3rded.,1999.

[22] B. FortzandM. Thorup,“Optimizing OSPF/IS-ISweightsin achangingworld,” To appearin IEEEJSAC SpecialIssueonAdvancesin Fundamentalsof NetworkManagement, Spring2002.

APPENDIX

A. MATLAB SOURCECODE% weighted least-squares estimate of the TM% Input:% A matrix A in constraints x=A*t% x vector x in constraints x=A*t% tg initial gravity model solution% w weight vector% Output:% t estimated traffic matrix (as a vector)% that minimizes |(t-tg)./w|% among all t’s that minimize |A*t-x|function [t] = wlse(A,x,tg,w)

% equivalently transform x=A*t into% xw=Aw*tw, where tw=(t-tg)./wxw = x - A*tg;[r, c] = size(A);Aw = A .* repmat(w’, r, 1);% solve tw=Aw*tw by computing the pseudo-% inverse of matrix Aw (through svd)tw = pinv(full(Aw)) * xw;% transform tw back to tt = tg + w .* tw;