IP Traffic Measurement: Technologies, Tools, and Protocols Jürgen Quittek NEC Europe Ltd., Network...

IP Traffic Measurement: Technologies, Tools, and

Protocols

Jürgen QuittekNEC Europe Ltd., Network Laboratories, Heidelberg, Germany

[email protected]

© NEC Europe Ltd., 2002Network Laboratories, Heidelberg

2

Outline

• Applications requiring traffic measurement

• General traffic measurement process

• Tools

• Protocols and Standards


3

Applications (1)Requiring Traffic Flow Measurement

• Usage-based accounting– input to charging and billing– various business model

• time-based, volume-based, QoS class-based• per application, per user, per user group

• Traffic engineering– optimizing network usage– traffic analysis on congested links

• origin of traffic• type of traffic• dynamic behavior (bursty, adaptive, …)

• Traffic profiling


4

Applications (2)Requiring Traffic Flow Measurement

• QoS monitoring– (passive) measurement of QoS properties– validating Service Level Agreements

• Attack detection and analysis– detecting (high volume) traffic patterns– investigation of origin of attacks

• Intrusion detection– detecting unexpected or illegal packets

• …


5

Classification &Flow Recording

Store(TCPdump)

ObservationPoint

The Traffic Measurement Process

PAYLOAD HEAD PAYLOAD HEAD


PacketCapturing

Filtering

Display(Ethereal)

Sampling

Transport

Conversion

Visualize(FlowScan)

Integrate into TE, attack detect., QoS monitoring,

accounting, ...

… other …

Optional:traffic

generation


6

IP Flow Definition

• “A flow is a set of packets with a set of common packet properties.”

• Application level flow versus flow monitored at a single observation point– between endpoints <--> at one or more obs. points– using same path <--> using different paths– end-to-end packets only <--> also dropped packets

• Uni-directional <--> bi-directional

• typical case: separation by 5-tuple– IP addresses, transport type, port numbers


7

Observation Points

• Shared Media

– shared wire Ethernet/Token Ring: OK– Ethernet with HUB: OK– Ethernet with switch: only broadcasts– Radio networks: not reliable

• Point-to-point– Capturing only on

end points or with splitter

sender receiver probe

sender receiver

probe


8

Packet Capturing at Routers

• Capturing on central CPU– observation point is

complete router– typically SW solution– not scalable

• Capturing on line card– restricted observation

point– typically hardware

support– scalable

line card

line card

line card

CPU

…


9

Packet Capturing and Filtering Technology: PCAP

• Library libpcap available on almost all Unix systems– creates copies of packets (up to a specified offset) in kernel

spaces– delivers copies to user space by callback functions– includes kernel space packet filter BPF (Berkeley Packet Filter)– filter specified by user, compiled by libpcap, transferred into kernel– commonly used: TCPdump, NeTraMet, … – native in BSD systems– Linux, AIX, Solaris, HP-UX have compatible kernel-level and/or

user-level implementations• sometimes with restricted functionality

• For probe: network interface card in promiscuous mode


10

Packet Capturing, Flow Recording and Transport Technology: NetFlow

• Developed by Cisco• De-facto standard• Available for (almost) all Cisco & Juniper router products• Dedicated probes available• Implementations on central CPU or line card• Packet capturing and flow recording with hardware support on

line cards• Measures all 5-tuple flows at a line card or at the entire router• Exports flow records using NetFlow protocol: simple records

sent over UDP • Supported by a huge variety of tools receiving NetFlow records


11

CAIDA Tools

• Developed and supported by CAIDA at University of California at San Diego: http://www.caida.org/tools/– cflowd – RTG– skitter– NeTraMet– CoralReef– Beluga


12

CAIDA Tools (2)

• cflowd– flow analysis tool currently used for analyzing NetFlow records– collections, storage, and basic analysis modules – data collection and analysis for capacity planning, trends

analysis, and characterization of workloads

• CoralReef– software suite collecting and analyzing data from passive

Internet traffic monitors– in real time or from trace files– Realtime monitoring via

• libpcap

• high-speed fiber network interface cards


13

CAIDA Tools (3)

• NeTraMet– open-source implementation of the IETF RTFM

architecture for Network Traffic Flow Measurement

• RTG– flexible, scalable, high-performance SNMP statistics

monitoring system. – collects time-series SNMP data from a large number of

targets quickly.– uses data base – includes utilities that generate configuration and target

files, traffic reports, 95th percentile reports and graphical data plots (supporting web-based interfaces).


14

CAIDA Tools (4)

• skitter– actively probing the Internet in order to analyze

topology and performance.• measures forward IP paths hop by hop• measures round trip time (RTT)• visualizes network connectivity

• Beluga– provides a real-time graph of RTTs and packet

loss to an end host– total round trip time and per-hop round trip time


15

More Tools

• See a long list of (NetFlow-related) tools at– http://www.switch.ch/tf-tant/floma/software.html

• FlowScan– analysis and nice graphical reporting of NetFlow input– http://net.doit.wisc.edu/~plonka/FlowScan/

• National Internet Measurement Infrastructure (NIMI) – http://ncne.nlanr.net/nimi/

• ntop– shows current network usage (like Unix ‘top’ program– http://www.ntop.org/ntop.html


16

Transport of Flow Records

• Requires inter-operation between sender and receiver• Standardization desirable

– de-facto standard NetFlow has some problems• IETF Standards

– RTFM (Meter MIB) • Real-Time Flow Measurement

– IPFIX (in progress)• IP Flow Information eXport

– PSAMP (in progress)• Packet Sampling


17

IPFIX Scope and General Requirements

• Goal: Find or develop a basic common IP Traffic Flow measurement technology to be available on (almost) all future routers

• Fulfilling requirements of many applications• Low hardware/software costs• Simple and scalable• Metering to be integrated in general purpose IP routers

and other devices (probes, middleboxes)• Data processing to be integrated into various

applications• Interoperability by openness

or standardization


18

IPFIX Requirements (1)

• Distinguishing flows by 5-tuple– IP addresses, transport type, port numbers– Supporting MPLS, DiffServ– Going on to more flexible flow definitions– Flexible aggregation of flows

• Metering Process– Reliability– Timestamps, time synchronization– Flow timeouts– Overload behavior

• sampling, simplifying, stopping


19

IPFIX Requirements (2)

• Data Export– Information model

• many header fields and statistics required• anonymization?

– Data model• flexible, extensible

– Data Transfer• reliability• security• congestion awareness• push and pull model reporting?• regular reporting interval• notification on specific events

• Configuration


20

Application

Collector

IPFIX Architecture Overview

Flow RecordProbe

(meter)

Observation Point

Flow Information Export





Exporter


21

IPFIX Scenarios

OM

E

Probe

OM

E

Simple Router

O OO OM

E

Complex Router

O OO OMO OO O

M

E

Multiple Exporters

O OO OM

E

O OO

OM

E

Protocol Converter

(Meter MIB)

OM

E

OM

E

OM

E

M E

Concen-trator

C E

Proxy

C …


22

Current State of IPFIX Standardization

• Requirement specification complete

• Protocol Selection in progress– no new protocol development– selection of an already existing protocol or

of a protocol contributed externally

• Elaboration / improvement of selected protocol will be last step before standardizing it


23

Existing Technologies

• IETF standards– RTFM– RMON, RMON2

• Proprietary technologies– NetFlow (Cisco)– sFlow (InMon)– LFAP (Riverstone)– Crane (XACCT)– …


24

Real-Time Flow Measurement (RTFM)

• Very flexible and powerful meter– programmable rule sets– can serve several readers– programmable overload behavior

• Reader polls meter

• Realization by SNMP Meter MIB

• Free software implementation NeTraMet

• No acceptance at manufacturers

• Complicated to use (too powerful)

• Specified by RFCs 2720 - 2724

Meter

Manager

Reader

Application


25

Remote Network Monitoring MIB

• Very flexible and powerful• Serves more general goals (analysis on layers 2-4)

• Just a monitoring tool, no measurement architecture defined

• Suited for very specific analysis tasks• High (hardware) performance requirements• Too complicated and too expensive for massive

usage in routers• Specified by RFCs 2021(RMON2), 2613,

2819(RMON), 2895, 2896, 3144


26

Router

NetFlow

• Proprietary by Cisco, but de-facto standard• Fast and efficient, implemented for IOS• Configurable measurement per 5-tuple• Unreliable (measurement & data transport)• Hardware-supported on some models• Not well documented

– re-engineered by Juniper• Versions 1-7

– fixed data model• Version 9 (under development)

– data model templates– optional reliable transport

Meter

Data collector

Application


27

Data collector

sFlow

• By InMon Corporation• Includes metering and data transmission

• Probabilistic sampling at meter

• Packet sampling and counter sampling

• Timestamping by data collector

• Configuration by sFlow MIB

• Poorly documented by informational RFC 3176

• Not adapted yet by other vendors

sMon Meter

Application


28

LFAP• Light-weight Flow Accounting Protocol• Proprietary by Riverstone (Cabletron)• Just data transfer protocol• Meter at Connection Control Entity (CCE) communicates to

Flow Accounting Server (FAS)• Tight and reliable interaction

between CCE and FAS• Reliable data transport• Flexible TLV coding of transferred data• Larger overhead than NetFlow • More cost-intensive at meter/CCE

and at data collector/FAS• See <draft-riverstone-lfap-00.txt>

CCE

Application

FAS


29

CRANE

• Common Reliable Accounting for Network Element (CRANE) Protocol

• Proprietary by XACCT• Just data transfer protocol

• Template-based data model

• Focus on reliability

• Not yet in extensive commercial use

• See <draft-kzhang-crane-protocol-02.txt>


30

IETF PSAMP Working Group

• Established in Summer 2002• Focus on sampling and capturing packets and on

transferring them to data collectors• Target applications

– traffic profiling– monitoring network behavior

• Closely related to IPFIX• Defines packet sampling with much more detail

– developing packet filtering and sampling information model– includes standardization of meter configuration

• Hot Issue: (partial) export of payload

IP Traffic Measurement: Technologies, Tools, and Protocols Jürgen Quittek NEC Europe Ltd., Network...

Documents

Transcript of IP Traffic Measurement: Technologies, Tools, and Protocols Jürgen Quittek NEC Europe Ltd., Network...