I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

TransPAC2 Measurement

John HicksTransPAC2

Indiana UniversityJhicks@iu.edu

APAN Conference – Singapore 19-July-2006

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

Overview

• TransPAC2 measurement goal• Arbor system measurement stats• Arbor system portal• Arbor system WSDL• Arbor system SOAP interface• 10G PC plans• Questions

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

TransPAC2 measurement goals

• TP2 is funded by the NSF and we are required to provide accounting of how TP2 is facilitating international scientific cooperation.

• TP2 is using Arbor Networks Peakflow SP system statistics and reporting capabilities for flow, BGP and security analysis.

• TP2 has three basic measurement goals:

• Provide public access to general statistics and data concerning the TP2 network link.

• Provide access to private data to trusted groups or individuals using the TP2 link.

• Provide a HP testing facility for application and network researchers.

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

Peakflow SP system

• The Arbor systems Peakflow SP system is a traffic analysis engine the works on a collection of routers.

• By default, the collection of routers is aggregated into a one network.

• General queries (no filters) return summaries of the entire network.

• Query filters provide one means of narrowing return data.

• Other scoping mechanisms are available.

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

Peakflow SP queries

• The Peakflow system collects flow and BGP data and provides a facility to query and filter desired data for a particular time slice.

• Peakflow SP is designed around XML queries.

• There is currently a limit of two filters for each query.

• Desired data can be further scoped by different query types.

• Example: filter on entity 1 (entity 1 could contain multiple routers) data, then on TCP ports (port #1, port#2, port#3, …)

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

Peakflow SP query filter types

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

Peakflow SP router query types

• Router query types include the following:

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

Peakflow SP Raw Flow queries

• Raw flow query types include the following:

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

Peakflow SP BGP queries

• BGP queries include:

• Diff - Reports a list of BGP change reports

• Raw - Reports a list of raw routes

• Summary - Reports the summary of BGP changes matching a filter

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

Peakflow SP

• The Peakflow SP system can return graphs and raw data for each type of query.

• Queries can be automatically run at schedules times (like cron).

• Query reports can be email to individuals or groups (including graphs and data).

• Email example:

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

• Peakflow SP offers a customer facing portal that provides access to some of the SP data.

• Portal data views are scoped to a subset of the systems data.

• Portals are a good way to provide private access to costumer data.

• One problem with portals is that scoping data is very course.

• For example: If a costumer sees an anomaly (large traffic from /24) in the data from a query and determines that it is coming from an interface not in the costumers scope. Further investigation is prohibited. If the system provides access to this interface then all interfaces are available.

Peakflow SP portals

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

• To solve this problem, we are using the Peakflow wsdl to provide more refined scoping of data.

• The Peakflow SP wsdl provide the following:

• getAlertGraph - For a given alert id, returns a graph of the total alert traffic per customer interface over the life of the alert.

• sqlQuery - Returns an SQL query in XML format.

• getTrafficData - Returns detailed sample data for items matching the query. Data is returned XML format.

• getTrafficGraph - Returns a graph of the data items matching the supplied query parameters.

Peakflow SP WSDL/SOAP

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

• getAlertSummaries - Returns summary information about the most recent count alerts, starting at offset alerts from the most recent alert. The optional filter can specify the name of a customer managed object.

•

Peakflow SP WSDL/SOAP

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

• getAlertInterfaces - Returns a detailed listing of all routers interfaces involved in the specified alert.

Peakflow SP WSDL/SOAP

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

• getAlertInterfaceDetails - Returns detailed information about router interfaces involved in the specified alert.

• getAlertInterfacesXML - Same as getAlertInterfaces but in XML format.

• getAlertRouterInterfacesXML - Same as getAlertRouterInterfaces but in XML format.

• getReport - Returns multiple graphs in one tar.gz file.

• getAlertStatisticsRaw - Returns raw statistics about requested alert.

Peakflow SP WSDL/SOAP

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

• usage: soap_client.py -c command <-s script|-h host> [-z zone_secret] [-o key1=value1 -o key2=value2 ...] [-w key1:file1 -w key2:file2 ...] • example: soap_client.py -h 'arbor.ren-isac.net' -z ’zonesecret' \ -c 'getTrafficData' -o query=@binby_router.xml \ -w data:results_router.xml

Python SOAP client

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

<?xml version="1.0"?><peakflow version="1.0"> <query id="query1" type="traffic"> <time start_ascii="1 hour ago" end_ascii="now"/> <unit type="bps"/> <search limit="200"/> <filter type="router" binby="1"> <instance name="observatory-gw.transpac2.net"/> </filter> <filter type="tcp_port"> <instance name="http"/> <instance name="smtp"/> <instance name="443"/> </filter> </query></peakflow>

query=@binby_router.xml

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

<?xml version="1.0"?><peakflow version="1.0" release="3.4.2"> <query id="query1" type="traffic"> <time start_ascii="1 hour ago" end_ascii="now"/> <unit type="bps"/> <search limit="200"/> <filter type="router" binby="1"> <instance name="observatory-gw.transpac2.net" value="223"/> </filter> <filter type="tcp_port"> <instance name="http" value="80"/> <instance name="smtp" value="25"/> <instance name="443" value="443"/> </filter> </query> <query-reply>

query=@binby_router.xml

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

<status_summary time="0.46"/> <status type="success" pid="19916" collector="collector" time="0.41" bytes="33354" items="3"/> <time start="1149787452" end="1149791052" start_ascii="06/08/2006 17:24:12 +0000" end_ascii="06/08/2006 18:24:12 +0000"/> <sample_info duration="300" count="12"/> <item id="223" name="observatory-gw.transpac2.net"> <key id="223" name="observatory-gw.transpac2.net"/> <class type="in"> <current value="43563000"/> <avg value="39963548"/> <max value="44114000"/> <pct95 value="44114000"/> <sample value="42611000|36259000|44114000|39776000|42468000|31353000|42009000|37840000|38385000|43563000|41221000|0"/> </class> <class type="out"> <current value="43563000"/> <avg value="39963548"/> <max value="44114000"/> <pct95 value="44114000"/>

query=@binby_router.xml (cont.)

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

• Peakflow SP currently provides a single “Zonesecret” to access the system remotely via SOAP.

• Scoping of data is done on the “proxy” server.

• Web presentation is currently done with PHP but other technologies such as AJAX are being explored.

• DB backend also under investigation.

• Proxy server code will be made available once private zonesecret can be secured.

• Costume interfaces are easily rolled out for private data.

Customer “Peakflow Proxy”

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

Client side

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

Client side

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

Client side

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

Client side

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

Client side

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

HP measurement machine

• TransPAC2 recently purchased a 10G capable machine with the following specifications:

• This machine will be use to test the TransPAC2 (Tokyo XP, NICT/KDDI) link and into Abilene (HOPI nodes).

• Further testing will continue as more 10G resources become available.

• Installation should be completed by August 2006.

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

Questions or Comments

John HicksIndiana University

jhicks@iu.edu