Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
111© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
2© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Advanced Performance Measurement with Cisco IOS® Service Assurance Agent
Session NMS-4041
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
3© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Objective of this Presentation
444© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Objective of this Presentation
• Give a clear understanding of SA Agent features and capabilities
• Understand the internals of SA Agent
• Be able to configure various probe types
• Advanced topics (performance, scalability,…)
• Some deployment recommendations
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
555© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
This Presentation Is Not About…
• Designing your network to deploy tight SLA services
• QoS configuration or recommendations
• We will talk about SA Agent only—not the various network management applications to use on the back-end
666© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Prerequisites
• Before attending this session, you should:Be familiar with SA Agent
Understand the SLA concepts
Know the basic concepts of network management
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
7© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Network Disturbance(When Theory and Practice Diverge)
888© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Murphy’s Law
• If anything can go wrong, it will
• If anything just cannot go wrong, it will anyway
• Left to themselves, things tend to go from bad to worse
• If everything seems to be going well, you have obviously overlooked something
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
999© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
SLA Components
• Delay
• Jitter
• Bandwidth
• Availability/connectivity
• Packet loss
• Out of Sequence (OoS)
• [Add your favourite here]
101010© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Latency (Delay)
• Propagation delay: the time it takes to the physical signal to traverse the path; (add 6 ns per meter for fibre, ie 36 ms for a transatlantic 6000 km link)
• Serialization delay is the time it takes to actually transmit the packet; depends on the bit-rate
• Queuing delay is the time a packet spends in router queues; depends on queue length and type
• Comfortable human-to-human audio is only possible for round-trip delays not greater than 100ms
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
111111© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Jitter
• This is the variation of the delay, a.k.a the ‘latency variance,’ can happen because:
• Variable queue length generates variable latencies
• Load balancing with unequal latency
• Harmless for many applications but real-time voice and video
121212© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Packet Loss
•• Loss of one of more packets, can happen Loss of one of more packets, can happen because…because…
•• CRC errorCRC error
•• Full queue (tail drop) or out of contractFull queue (tail drop) or out of contract
•• Route change (temporary drop) or Route change (temporary drop) or blackholeblackhole route (persistent drop)route (persistent drop)
•• Interface or router downInterface or router down
•• MisconfiguredMisconfigured accessaccess--listlist
•• ……
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
131313© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Misordering [1/2]
• This is not a rare situation…
• According to a study, roughly 25% of the hosts monitored on the Internet exhibit reordering
• For the hosts that exhibited reordering on average 8 of the 50 packets were identified as being out of order
(Results Are Based on “Packet Reordering Is Not Pathological Network Behavior, Jon C. R. Bennett, Craig Partridge and Nicholas Shectman, IEEE/ACM Transactions on Networking, Vol. 7, No. 6, December 1999, p789”)
141414© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Misordering [2/2]
• Out-of-order packet delivery, can happen because…
• Load balancing through multiple paths having different latencies
• Typically happening on parallel architectures (equivalent to multiple parallel routers)
• …
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
151515© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
But Also…
• Packet alteration—the content is randomly modified
• Packet duplication—the same packet arrives multiple times (generally combined with misordering)
161616© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
To Summarize…
•• The network is like a live ecosystemThe network is like a live ecosystem
•• There are harmless and harmful species There are harmless and harmful species living togetherliving together
•• They cannot always be under controlThey cannot always be under control
•• But at least we can vigilantly observe But at least we can vigilantly observe what’s going onwhat’s going on
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
17© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
SAA Overview
181818© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Current Solutions to Measure SLAs?
• Wait for problem to happen, and customer to complain
Reactive approach
• Manually Monkey approach
• Custom, home-made application The geeky approach
• Special hardware probes The expensive approach
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
191919© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Current Solutions Drawbacks
• Requires additional hardware
• New software, protocols
• Additional configuration skills
• Eventually adding a new vendor, support contract…
202020© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
The Idea behind SAA
• If you have a running Cisco IOS router, turn it into a probing device
The smart approach
• Reuse your current equipment and enhance existing network management applications
(ex: CiscoWorks, VPNSC, Infovista, Concord eHealth, Agilent Firehunter…)
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
212121© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Supported Cisco IOS Version
XXXXSNMP Support
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
12.2(2)T
XAPM
XICMP Path Jitter
XFrame-Relay (CLI)
XMPLS/VPN Aware
XXFTP Get
XXUDP Jitter with One Way Latency
XXXDLSw+
XXXDHCP
XXXDNS
XXXHTTP
XXXUDP Jitter
XXXXTCP Connect
XXXXUDP Echo
XXXXSSCP(SNA)
XXXXXICMP Echo Path
XXXXXICMP Echo
12.2(11)T(Eng2)
12.1(1)T12.2
12.0(5)T12.0(8)S
12.0(3)T11.2Feature/Release
222222© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Measurement Capabilities
HTTPHTTPDLSwDLSw
JitterJitterPathEchoPathEcho
EchoEchoDNS/DHCP
FTPFTP
Increasing Service ValueIncreasing Service ValuePath JitterPath Jitter
ConnectConnect
EchoEcho
SNASNA
Cisco IOS-BasedService Assurance*
Agent
TCP
QoS Support
MPLS VPN Aware
Frame Relay
APM
UDP ICMP
*With Cisco IOS 12.2(9)T
(TOS)(TOS)
ATM*
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
232323© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Availability
Cat5K
Cisco GSR, 10K
Cisco 6400/7200/7500,uBR7200
Cisco 800/100x/14xx/16xx/17xx
Cisco 25xx/26xx
MC3810 Cisco 36xx
Cisco AS5300/5800
Cisco 4500/4700Catalyst
5K/6K with RSM/MSFC
All Cisco IOS-Based Platforms
242424© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
SA Agent History
• Used to be called RTR, renamed SA Agent in 12.0(5)T; we call it Engine 1
• Initially only for ICMP Echo
• New Engine 2 introduced in 12.2(11)T, and will be present in all 12.3 trains
Major rewrite of the SAA code
Faster and more scalable
Memory usage reduced by a factor 2 to 5
ATM and Frame Relay L2 probes
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
252525© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
To Summarize…
• Wide measurement capabilities(UDP, TCP, ICMP…)
• Millisecond precision (do not use Cisco IOS timers)
• Accessible using CLI and SNMP• Proactive notification via SNMP traps• Already in Cisco IOS—available on most
platforms at no additional cost• All IP interfaces supported, physical
and logical
26© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
SAA Architecture
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
272727© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Network
Global Architecture Overview
SA Agent
Responder
Responder
IP Server
Management
Targets
Source
SNMP/CLI
Probe
282828© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
SAA Sender
• Cisco IOS box that sends probes
• Where the probes are configured
• Where all the results are calculated and stored
• Target might be another entity running Cisco IOS, or another system like a server
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
292929© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
SAA Responder
• Runs on Cisco IOS• To activate, add ‘rtr responder’ to the
config, or set rttMonApplResponder.0=1 with SNMP
• Sender uses the SAA control protocol to communicate with responder before sending the test packets
• Responder knows the type of operation, the port used, the duration
• Communication on UDP 1967 and can be authenticated with MD5, not encrypted
303030© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
SAA Operation with Responder [1/2]
SAA Sender SAA ResponderControl Message Ask Receiver to
Open Port 2020 on UDP)
Responder Says OK
Sending Test Packets…
Start Listening onUDP Port 2020
UDP, 2020
Done: Stop Listening
ControlPhase
ProbingPhase
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
313131© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
SAA Operation with Responder [2/2]
• The responder, based on the type of operation, may insert in/out timestamps in the packet’s payload
• Processing time spend on the responder can therefore be calculated and deduced
• The response time is always calculated by the sender
323232© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
MD5 Configuration Example
key chain saa_kckey 1key-string key
rtr key-chain saa_kc
04:02:24.271: responder receives request04:02:24.271: Ver: 1 ID: 22 Len: 52 04:02:24.275: process port enable clsd04:02:24.275: cmd: command: , ip: 10.52.132.68, port: 6666,
duration: 5200
17:01:28.745: RTR 1: Starting An Echo Operation - IP RTR Probe 117:01:28.745: source=10.52.132.69(52653) dest-ip=10.52.132.68(6666)17:01:28.745: sending control msg:17:01:28.745: Ver: 1 ID: 21 Len: 52 17:01:28.749: cmd: command: RTT_CMD_JITTER_PORT_ENABLE, ip: 10.52.132.68,
port: 6666, duration: 520017:01:28.757: receiving reply17:01:28.761: Ver: 1 ID: 21 Len: 8
SAA Sender SAA Responder
RTT_CMD_JITTER_PORT_ENABLERTT_OK
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
333333© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
SAA Control—Scanning
• SAA responder can be detected with a port scanner and generates an “RTR responder: bad format” when debug RTR error is enabled
• On the top of MD5, configure an access-list to restrict it’s access if it runs on a public network
linux-f7-1:~ # nmap -sU 10.52.132.68 -p 1967Starting nmap V. 3.00 ( www.insecure.org/nmap/ )Interesting ports on c26f7-11.nsite.cisco.com (10.52.132.68):Port State Service1967/udp open unknownNmap run completed -- 1 IP address (1 host up) scanned in 1 second
34© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
SA Agent ConfigurationVia Command Line (CLI)
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
353535© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Configuring an Operationc26f7-12(config)#rtr 1
c26f7-12(config-rtr)#type ?
dhcp Perform DHCP Operation
dlsw Perform DLSw Keepalive Operation
dns Perform DNS Query
echo Perform Point to Point Echo Operations
ftp Perform ftp operation
http Perform HTTP Operations
jitter Perform Jitter Operation
pathEcho Perform Path Discovered Echo Operations
tcpConnect Perform TCP Connect Operations
udpEcho Perform UDP Echo Operations
363636© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Parameters
router (config-rtr)#?
frequency Operation Frequency Value in Seconds
lsr-path Loose Source Routing Path
owner Owner of Entry
request-data-size Requested Request Payload Size
response-data-size Requested Response Payload Size
tag User Defined Tag
threshold Operation Threshold in msec
timeout Operation Timeout Value in msec
tos Type Of Service
verify-data Verify Data
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
373737© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
ICMP Echo Operation
• Target can be any IP host
• Round-trip time computed by measuring the time taken between sending an ICMP Echo request message and receiving the reply
• Processing delays on the source router only is subtracted
383838© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
ICMP Echo Operation (Measurement)
• The round trip time is T = T2-T1
• The processing time spend on the target host cannot be measured, nor predicted, so it will be included in the total round trip time (see later)
• Because it might be inaccurate, use this probe for connectivity measurement (check link connectivity, if a server is still online, if a dial-on-demand line is up,…)
T1
T2T3 Target HostSAA
Sender
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
393939© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
ICMP Echo Operation (Example)
rtr 1type echo protocol ipIcmpEcho 10.32.130.2tos 0x20frequency 120
rtr schedule 1 life forever start-time now
404040© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
ICMP Echo Operation (Output)
c26f7-12#sh rtr op 1Current Operational State
Entry Number: 1Modification Time: 11:28:21.000 CET Thu Aug 22 2002Diagnostics Text: Last Time this Entry was Reset: NeverNumber of Octets in use by this Entry: 1490Connection Loss Occurred: FALSETimeout Occurred: FALSEOver Thresholds Occurred: FALSENumber of Operations Attempted: 1Current Seconds Left in Life: infinite - runs foreverOperational State of Entry: activeLatest Completion Time (milliseconds): 2Latest Operation Start Time: 11:28:21.000 CET Thu Aug 22 2002Latest Operation Return Code: okLatest 10.52.130.2
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
414141© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
SAA Accuracy…ICMP Echo Probe
• With unloaded receiver, SAA measures 1.5 ms
• With high CPU load on the receiver: 45 ms!!
ICMP Echo Probe
Any System Will Report Wrong Results when Too Much CPU Time Is Spent on the Receiver between
the ICMP Echo Request and Echo Reply
Fortunately, We Have a Solution…
(90% Process Load)
ResponderSender
424242© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Processing Time Measurement
• When running the responder, we have a clear advantage, because…
• There is a mechanism to evaluate the processing time spend on the receiving router
• Insert a timestamp when the responder receives the packet, and when it replies
• Receive timestamp done at interrupt level, as soon as the packet is dequeued from the interface driver;absolute priority over everything else
• With SA Agent, this mechanism is implemented for both UDP Echo and UDP Jitter probes
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
434343© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
UDP Echo Operation
• Uses either well-known UDP port 7 (echo service)—or any other custom port
• Can run with or without the responder
• However, it requires the responder for more accurate results; the processing delay spend on both source and destination is measured and deduced from the total RTT
444444© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
T2
UDP Echo Operation (w/SAA Responder)
• We have no control on the queuing delay on the source and destination, but this is experienced by real traffic too, and must be accounted as such
T5
T4
T3
Processing Delay on the Source: Tps = T5-T4
Processing Delay on the Destination: Tpd = T3-T2
Round Trip Time Delay: T = […] = T2 - T1 + T4 - T3
Sender
T1
Responder
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
454545© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
UDP Echo Operation (Example)
rtr 1type udpEcho dest-ipaddr 10.52.132.68 dest-port 7threshold 200
rtr schedule 1 start-time now
464646© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
UDP Echo Operation (Output)
c26f7-12#sh rtr op 1Current Operational State
Entry Number: 1Modification Time: 13:55:05.000 CET Thu Aug 22 2002Diagnostics Text: Last Time this Entry was Reset: NeverNumber of Octets in use by this Entry: 1490Connection Loss Occurred: FALSETimeout Occurred: FALSEOver Thresholds Occurred: FALSENumber of Operations Attempted: 1Current Seconds Left in Life: 3595Operational State of Entry: activeLatest Completion Time (milliseconds): 2Latest Operation Start Time: 13:55:05.000 CET Thu Aug 22 2002Latest Operation Return Code: okLatest 10.52.132.68
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
474747© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
The SAA Responder Processing Delay Will Be Subtracted to the Final Results
The SAA Responder Processing Delay Will Be Subtracted to the Final Results
• With unloaded receiver: 1.5 ms
• With 90% CPU receiver: 1.8 ms
SAA Accuracy—UDP Echo Probe
UDP Echo Probe
ResponderSender
(90% Process Load)
484848© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
UDP Jitter Operation
• Measures the delay, delay variance (jitter) and packet loss by generating periodic UDP traffic
• Measures: per-direction jitter, per-direction packet-loss and round trip time
• Detect and report out-of-sequence and corrupted packets
• One-way delay requires Cisco IOS 12.2(2)T or laterand clock synchronization between source and destination
• One-way jitter does not require clock sync
• Always requires SAA responder
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
494949© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
UDP Jitter—Measurement Example
SAA RTx = Receive Tstamp for Packet x
Send Packets
ST2 ST1
P1P1P2P2P2i1
AT1 AT2
Reflected Packets
P2P2P1P1i4
Responder
Dx = Processing Time Spent between Packet Arrival and Treatment
IP Core
STx = Sent Tstamp for Packet x
With Each Packet Is Associated STWith Each Packet Is Associated STxx, RT, RTxx, AT, ATxx, d, dxx——so so the Source Can Now Calculate:the Source Can Now Calculate:JitterSDJitterSD = (RT= (RT22--RTRT11))--(ST(ST22--STST11) =) = ii22--ii11JitterDSJitterDS = (AT= (AT22--ATAT11))--((RT((RT22+d+d22))--(RT(RT11+d+d11)) =)) = ii44--ii33
ATx = Receive Tstamp for Packet x
Receive Packets
RT2 RT1
P1P1P2P2P2i2
RT1+d1 RT2+d2
Reply to Packets
P2P2P1P1i3
505050© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
UDP Jitter Operation Jitter Computation
• If packets are sent with 10ms interval, positive jitter means they have been received with more than 10ms interval
• Negative jitter means less than 10ms interval
• Zero jitter means they are received with the same inter-packet delay (the variance is zero)
• Jitter should remain as low as possible for real-time traffic such as voice over IP
• No need to have clocks synchronized
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
515151© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
UDP Jitter Operation One-Way Delay Computation
• One-way delay measurement requires the clocks on source and target routers synchronized
• Use Network Time Protocol (NTP) server, eventually with GPS device as reference
• Use a GPS device on the auxiliary port of a 7200 (ex: Trimble Palisade GPS)
• If the time is not synchronized, SAA skips the one-way delay results; we tolerate a drift of 10% of the RTT; the shorter the delay, the stricter it will be
• GPS requires clear-sky view: not always feasible
• CDMA clocks works where a cell-phone work
525252© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Ack: (1,1)Ack: (1,1)
UDP Jitter OperationPacket Loss
Sender
Tim
e
Result:PacketLossSD = 1PacketLossDS = 1
Result:PacketLossSD = 1PacketLossDS = 1
Send Counter: 1
Responder
Rx: (5,4)Rx: (5,4)
R Missed a Packet (Only 4 Received while 5 Sent): PacketLossSD += 1
R Missed a Packet (Only 4 Received while 5 Sent): PacketLossSD += 1
Index of the ACKd Packet
Index of the ACKd Packet
Rx Counter: 1
Rx Counter: 2
Rx Counter: 3
Rx Counter: 4
Rx: (1,1)Rx: (1,1)
Rx: (3,3)Rx: (3,3)
XX Ack: (2,2)Ack: (2,2)
Ack: (3,3)Ack: (3,3)
Ack: (5,4)Ack: (5,4)
Send Counter: 3
XXSend Counter: 4
Send Counter: 5
Current RxCountCurrent RxCount
Send Counter: 2
R Received the Packet, but Did Not Received the ACK: PacketLossDS += 1
R Received the Packet, but Did Not Received the ACK: PacketLossDS += 1
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
535353© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
rtr 1type jitter dest-ipaddr 10.52.130.68 \
dest-port 3456 num-packets 20
rtr schedule 1 start-time now
UDP Jitter Operation (Example)
• Simple example:
UDP Jitter Probe to 10.52.130.68, Port 3456
Send 20 Packets Each Time
545454© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
UDP Jitter Operation (Example)
• Simulating G.711 VoIP call• Use RTP/UDP ports 16384 and above, the packet size is
200 bytes (160 bytes of payload + 40 bytes of header)• Packets are sent every 20 milliseconds• Marked with DSCP value of 8 (TOS equivalent 0x20)
rtr 1 type jitter dest-ipaddr 10.52.130.68 dest-port 16384 \
num-packets 1000 interval 20tos 0x20frequency 60request-data-size 200rtr schedule 1 life forever start-time now
AB C
A = 20 msB = 20 s (1000 x 20 ms)C = 40 s (60 s – 20 s)
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
555555© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
etychon-vpn#sh rtr op 1Current Operational State
Entry Number: 1Modification Time: 08:22:34.000 PDT Thu Aug 22 2002Diagnostics Text: Last Time this Entry was Reset: NeverNumber of Octets in use by this Entry: 1594Number of Operations Attempted: 1Current Seconds Left in Life: 574Operational State of Entry: activeLatest Operation Start Time: 08:22:34.000 PDT Thu Aug 22 2002Latest Oper Sense: okRTT Values:NumOfRTT: 997 RTTSum: 458111 RTTSum2: 238135973Packet Loss Values:PacketLossSD: 3 PacketLossDS: 0PacketOutOfSequence: 0 PacketMIA: 0 PacketLateArrival: 0InternalError: 0 Busies: 0(cont…)
UDP Jitter Operation (Output) [1/3]
Average RTT Was 458111/997 = 459ms
3 Packets Lost S->Dout of 1000 Sent
565656© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
(…cont)Jitter Values:MinOfPositivesSD: 1 MaxOfPositivesSD: 249NumOfPositivesSD: 197 SumOfPositivesSD: 8792 Sum2PositivesSD: 794884MinOfNegativesSD: 1 MaxOfNegativesSD: 158NumOfNegativesSD: 761 SumOfNegativesSD: 8811 Sum2NegativesSD: 139299MinOfPositivesDS: 1 MaxOfPositivesDS: 273NumOfPositivesDS: 317 SumOfPositivesDS: 7544 Sum2PositivesDS: 581458MinOfNegativesDS: 1 MaxOfNegativesDS: 183NumOfNegativesDS: 603 SumOfNegativesDS: 6967 Sum2NegativesDS: 336135Interarrival jitterout: 16 Interarrival jitterin: 35One Way Values:NumOfOW: 0OWMinSD: 0 OWMaxSD: 0 OWSumSD: 0 OWSum2SD: 0OWMinDS: 0 OWMaxDS: 0 OWSumDS: 0 OWSum2DS: 0
UDP Jitter Operation (Output) [2/3]
Follow RFC1889 (RTP) to Measure Jitter with Noise Reduction
No Synchro between Clocks: All ZeroesNo Synchro between Clocks: All Zeroes
Source to Destination Jitter
Destination to Source Jitter
See Next Slide
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
575757© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
MinOfPositivesSD: 1 MaxOfPositivesSD: 249
NumOfPositivesSD: 197 SumOfPositivesSD: 8792 Sum2PositivesSD: 794884
MinOfNegativesSD: 1 MaxOfNegativesSD: 158
NumOfNegativesSD: 761 SumOfNegativesSD: 8811 Sum2NegativesSD: 139299
UDP Jitter Operation (Output) [3/3]
Smallest Positive Jitter
Number of Packets with a Positive Jitter
Sum of All Positive Jitter
Sum the Squares of All Positive Jitter
Sum the Squares of All Negative Jitter
Smallest Negative Jitter
Number of Packets with a Negative Jitter
Biggest Positive Jitter
Biggest Negative Jitter
Sum of All Negative Jitter
585858© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
UDP Jitter Operation—Calculate Jitter
• There is no average jitter on SAA output
• You can calculate it with:
NumOfRTT
DS}|SDNegative}{|tiveSumOf{Posi∑=AvgLat
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
59© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
SA Agent Configuration(Using SNMP—Quick Overview)
606060© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
SAA Configuration via SNMP
• Uses the RTTMON MIB• For all the probes should be set at least:
rttMonCtrlAdminRttTypeThe type of SAA operation to be performed; this value must be set in the same PDU or before setting any type specific configuration
rttMonEchoAdminProtocolSpecifies the protocol to be used to perform the SAA operation; the following list defines what protocol should be used for each probe type: echo, pathEcho, ipIcmpEcho, udpEcho, ipUdpEchoAppl, tcpConnect, ipTcpConn http, httpAppl jitter, jitterAppl dlsw, dlswAppl dhcp, dhcpAppl ftp, ftpAppl
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
616161© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
SA Agent via SNMP (Example)
• SNMP set for ICMPEcho:rttMonCtrlAdminStatus.1 1
rttMonCtrlAdminRttType.1 1
rttMonEchoAdminProtocol.1 2
rttMonEchoAdminTargetAddress.1 "05 00 00 02"
rttMonEchoAdminTOS.1 5
rttMonScheduleAdminRttStartTime.1 1
rttMonScheduleAdminRttLife.1 200Last for 200 SecondsLast for 200 Seconds
Start NowStart NowTOS = 5TOS = 5
Destination IP = 5.0.0.2Destination IP = 5.0.0.2IpIcmpIpIcmp
Echo ProbeEcho Probe
Probe Is ActiveProbe Is Active
Probe Index = 1Probe Index = 1
626262© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
SNMP Configuration—More
• Additionally each probe requires specific variables to be set; see the RTTMON MIB for more details
• For the probe to be visible in the running configuration, and hence saved, you must set rttMonCtrlAdminNvgen to 1; by default, the value is 0
• A running probe cannot be changed; this is also valid for CLI (an exception is made for trigger admin variables)
• Not everything is configurable by SNMP, and not everything can be retrieved by SNMP; check the MIB and CCO documentation for details
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
63© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
SA Agent Options
646464© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
VRF—Awareness Issue (for MPLS/VPN)
• How to send a probe from the SA Agent to a specific VPN?
• By default, local processes are not executed in a VRF context
• Route lookup is done in the global routing table, and the wrong route is selected
10.10.10.110.10.10.1
10.11.10.110.11.10.1
10.12.10.110.12.10.1
SA Agent(PE)
CEs with VRFs Red, Blue and Yellow
We Need a Way to Execute a Probe in a VRF Context—E.G. Red, Blue or YellowWe Need a Way to Execute a Probe in a VRF Context—E.G. Red, Blue or Yellow
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
656565© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Solution—SAA for MPLS/VPN
• SAA probes are vrf-aware since 12.2(2)T
• Supported on ICMP Echo, ICMP Path Echo, UDP Echo and UDP Jitter probes
• Allows measurement from PE to anything
• Use vrf vrf-name option
666666© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
rtr 1type jitter dest-ipaddr 10.52.130.68 \
dest-port 3456vrf blue
rtr schedule 1 start-time now
SAA for MPLS/VPN (Example)
• Sends a probe to a remote SAA in the context of the blue VPN:
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
676767© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
TOS Marking
• Probes can be TOS marked
• Only TOS setting is supported, no diffServ(see next slide to perform translation)
686868© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Always Zero
Converting between TOS and DiffServ
128 64 32 16 8 4 2 1
32 16 8 4 2 1
554444176 (0xB0)176 (0xB0)101 100101 10011141456 (0x38)56 (0x38)001 110001 110
554040160 (0xA0)160 (0xA0)101 000101 000PrecedencePrecedenceDSCPDSCPToSToSBinaryBinary
Multiply by 4 Divide by 8
In Cisco IOS the 8 Bits of TOS Are Set from Right to Left
TOS(RFC795)TOS(RFC795)
DiffServ(RFC2474)DiffServ(RFC2474)
Precedence
4 2 1P2P2 P0P0P1P1 CUCU
D4D4 D3D3 D2D2 D1D1 D0D0DSCP (6 Bits)
D5D5
ToST3T3 T2T2 T1T1 T0T0
CUCU CUCU
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
696969© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Reaction Threshold [1/2]
martel(config)#rtr reaction-configuration <n> ?
action-type RTR Reaction Action Type
connection-loss-enable RTR Enable Connection LossReaction
threshold-falling RTR Falling Threshold Value
threshold-type RTR Reaction Threshold Type
timeout-enable RTR Enable Timeout Reaction
martel(config)#rtr reaction-trigger <probe> <targetProbe>
707070© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Reaction Threshold [2/2]
• Allows to set threshold on conditions (connection lost, threshold exceeded, single of multiple violations,…)
• Allows to set a reaction: send an SNMP trap, start another probe for problem diagnosis, or both
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
717171© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Low Water Mark
• Every time a probe is created, the router checks if there is more free memory than ‘LowWaterMark’; if not, the probe is not created
• By default, it’s set to 25% of the free memory on the router after bootup; if ‘0’, routers can create probes till run out of memory (not recommended)
etychon-vpn(config)#rtr low-memory ?
<0-4294967295> Low Water Memory Mark
727272© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Loose Source Routing (LSR) [1/2]
Source(207.139.2.10)
Destination(195.207.139.1)
Problem: If We Have Two Equal Paths, How to Measure One Specific Path?Problem: If We Have Two Equal Paths, How to Measure One Specific Path?
rtr 3type echo protocol ipIcmpEcho 195.207.139.1lsr-path 193.121.249.55rtr schedule 3 start-time now
Hop (193.121.249.55)
Solution: Specify a Hop with Loose Source Routing
Example:
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
737373© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Loose Source Routing (LSR) [2/2]
• To compute response time while the sender gives one or more hops that the packet must go through
• Valid for ICMP Echo probes• Intermediate devices might not support LSR,
which is often the case for security reasons• More processing time will be spend on the
intermediate hops than with regular probe; (LSR packets are sometimes process switched)
74© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
SAA Performance and Scalability
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
757575© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
SAA Performance Measured(Cisco 2600 Running SAA Engine 1)
14401440
39003900
20002000
14401440
20002000
# of Source # of Source Probe Probe
Operations Operations per Minuteper Minute
51.5351.5397K Total97K TotalResponder (UDP JitterResponder (UDP Jitter
49.449.458K Total58K TotalResponder (UDP Echo)Responder (UDP Echo)
6.336.3311K per Probe11K per ProbeICMP EchoICMP Echo
54.3154.3117K per Probe17K per ProbeUDP JitterUDP Jitter
28.8328.8313K per Probe13K per ProbeUDP EchoUDP Echo
Average CPU Average CPU Usage (%)Usage (%)
Average Average Memory Usage Memory Usage
(Bytes)(Bytes)TypeType
(Cisco 2600 Running SAA Engine 1)
767676© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
SAA Performance (Eng1 vs. Eng2, Jitter)
1200120010801080960960840840720720600600480480360360240240
Probes per Probes per MinuteMinute
38385757202034345151181830304646161627274040141423233434121220202727101016162020881313141466888844
CPU Load on CPU Load on Eng2Eng2
CPU Load on CPU Load on Eng1Eng1
Probes per Probes per SecSec
(Cisco 2600—12.2(8)T5 vs. Latest Eng. Build)
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
777777© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
SAA Performance (Eng1 vs. Eng2, Jitter)
8
20
34
46
57
8
16
23
30
38
0
10
20
30
40
50
60
4 8 12 16 20
Probes per Second
CP
U L
oad
(%
)
Engine 1Engine 2
(Cisco 2600—12.2(8)T5 vs. Latest Eng. Build
787878© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
SAA Performance—CPU Load by Platform
35
31
27
23
21
15
10
7
6
5
4
3
2
1
1
7200/2257200/225
57
56
45
41
40
34
29
25
16
9
6
4
3
1
1
37253725
36003600
33603360
31203120
28802880
26402640
24002400
21602160
19201920
16801680
14401440
12001200
960960
720720
480480
240240
Probes/Probes/MinuteMinute
95859963636
1464654040
1970714444
2276774848
2381825252
2595965656
44945772828
65652883232
276060
34239662424
33532572020
32827461616
32113341212
31272088
388844
7500/ 7500/ RSP8RSP8364036402650XM2650XM26002600Probes/Probes/
SecondSecond
(Jitter Probe Running Eng 1—500 Active Probes—Cisco IOS 12.2(8)T5)
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
797979© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
SAA Performance—by Platform(Jitter Probe Running Eng 1—500 Active Probes—Cisco IOS12.2(8)T5
0
20
40
60
80
100
120
4 12 20 28 36 44 52 60
Probes per Second
CP
U L
oad
(%
) 26002650XM36407200/22537257500/RSP8
808080© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Relation between Probes and Frequency
• Each operation’s results have to be stored into a hierarchical structure
• So, the processing time increase with the number of configured probes
• With the same amount of probes starting every second, the higher the configured probes, the higher the CPU
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
818181© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Probes/Frequency—Graphical View
01020304050607080
4 12 20 28 36 44 52 60
Probes per Second
CP
U L
oad
(%
)
500 Probes
1000 Probes
2000 Probes
Cisco 7200VXR /NPE-225 Running Eng1 12.2(8)T5
828282© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Probes/Frequency—Graphical View
Cisco 7500/RSP8/250Mhz Running Eng1 12.2(8)T5
0
20
40
60
80
100
4 12 20 28 36 44 52 60
Probes per Second
CP
U L
oad
(%
)
500 Probes
1000 Probes
2000 Probes
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
838383© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
SAA Memory Usage
< 3.2 KB< 3.2 KB< 17 KB< 17 KBICMP EchoICMP Echo
< 3.5KB< 3.5KB< 19 KB< 19 KBUDP EchoUDP Echo
< 12KB< 12KB< 24 KB< 24 KBUDP JitterUDP Jitter
Eng2Eng212.2(13)T12.2(13)T
Eng1Eng112.2(8)T512.2(8)T5
Engine 2 Reduce the Memory Usage by a Factor 2 to 5
848484© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
SAA—Scalability Recommendation on Memory Usage
• Do not use more than 50% of the router’s total memory for SAA
• This amount depends on which image and features you are running…up to you to fine tune
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
858585© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
SAA Accuracy—The Dilemma
• A router is, basically, a forwarding machine
• SAA is a time sensitive application running on a forwarding machine
• Cisco IOS processes uses a non-premptive scheduler
• This creates potential issues…but we have solutions
868686© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
SAA Accuracy—ICMP vs. UDP
• As seen before—for RTT accuracy, always use UDP Echo or jitter with SAA responder
• Only in this case, processing time spent on the sender and responder routers will be subtracted
• Results more accurate regardless of the sender and receiver CPU process load
• But…if we have a high CPU interrupt load, like packet forwarding on centralized platforms, things may change…
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
878787© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
UDP Echo Probe
ResponderSender
SAA Accuracy—Forwarding
Traffic Generator
Traffic Generator
(90% Fwd Load)
• With unloaded receiver: 1.5 ms
• With 90% CPU receiver, loaded by forwarded traffic: 10 ms!!
SA Agent Timestamping Routines Are in Competition with the Forwarded Traffic Done at
Interrupt Level Too
SA Agent Timestamping Routines Are in Competition with the Forwarded Traffic Done at
Interrupt Level Too
888888© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
SAA Accuracy—Forwarding Router
• SAA may be inaccurate on a router loaded with forwarded traffic
• Reason is that interrupt level code (ie: interface) is in competition with SA Agent
• Actual solution: use a dedicated, non forwarding router (called shadow router)
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
898989© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
SAA Accuracy—Test Results
• Tests have shown good accuracy if the router’s forwarding CPU load is below 30%. This is Cisco’s recommendation.
• Results become unrealistic when the forwarding CPU load reach the 60% utilization
• Process load has a negligible effect on UDP probes. Remaining at 60% process load is a comfortable value.
909090© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Accuracy with TOS-Marked Packets
• SAA probes can be sent with a specific Type of Service (TOS) value
• The right precedence will be applied when routing the packet, but what about the sending router?
• It depends…
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
919191© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Accuracy—Per Platform TOS Queuing
• Non-distributed platforms and 7500:Locally originated packets with proper TOS marking will go through the same outgoing queuing treatment; so SA Agent packets go through the corresponding WFQ queues
• For the Cisco 12K (GSR) and 10K (ESR):Queuing is done on the line cards; locally originated packets, like SAA probes, are all going to the default queue regardless of their original precedence; the default queue is typically slower
92© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Best Practice and Design Recommendations
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
939393© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Why Using a Shadow Router?
• If your PE is already well loaded, or…
• If your PE lacks memory, or…
• If your PE is a distributed platform, or…
• If you want to isolate SAA and routing, or…
• If you want to be able to upgrade the SAA engine without disturbing the network, then…
• Use a Shadow Router
949494© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
What Is a Shadow Router (SR)?
• Dedicated router to run the SAA engine, behave as a CE or a PE in a POP
• If behave as PE, member of VRFs but advertise no route
• If behave as CE, use one link per VRF; ideally use Ethernet with VLAN
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
959595© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Shadow Router as CE—Pros and Cons
• Cheap as any router from Cisco 800 will suffice
• Use one shadow per VRF, or set one VLAN per VRF [this requires Cisco 1700+ for dot1q]
• Cannot cope with overlap addresses
969696© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
dot1q
Shadow Router as CE—Example
Blue VPNBlue VPN
Green VPNGreen VPN
PE
Shadow Router(As a CE)
interface fastethernet 4/1.100encapsulation dot1q 100ip vrf forwarding blueip address …
interface fastethernet 4/1.101encapsulation dot1q 101ip vrf forwarding greenip address …
interface fastethernet 1/1.100encapsulation dot1q 100ip address …
interface fastethernet 4/1.101encapsulation dot1q 101ip address …
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
979797© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Shadow Router as PE—Pros and Cons
• Requires at least a Cisco 3600 for PE functionality, often the remote access router can be leveraged
• Works for all VRFs without touching the configuration
989898© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Blue VPNBlue VPN
Green VPNGreen VPN
Shadow Router as PE—Example
Evaluate PE-to-CE for Green VPN
Evaluate PE-to-PE, or PE-to-CE
for Any VPN
The Shadow Router Emulates a PE with VRF-Aware SAA
PE
Shadow
Evaluate PE-to-CE for Blue VPN
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
999999© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
How to Probe?
• Full mesh
• Partial mesh
• Composite SLAs
100100100© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Full Mesh
• No of probes is a square function of nodes
• Does not scale
NodesNodes ProbesProbes22 11
33 33
44 66
55 1010
66 1515
77 2121
88 2828
…… ……
100100 49504950
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
101101101© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Partial Mesh
San JoseAmsterdam
Raleigh
Brussels
Paris
London
• Full mesh is not always desirable• Select only critical path, like branch offices
to headquarters• Dramatically reduces the number of probes
102102102© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Composite SLA for Delay
A
B
C
D
E
F G H
10 ms
8 ms
12 ms
5 ms
11 ms10 ms
7 ms
C->F = 24 msC->F = 24 ms
• Total delay can be easily calculated by adding the measured delay
• For n nodes, full mesh requires n(n-1)/2 probes while composite requires n-1 probes
• Measurement is less accurate, as each measurement carry it’s own error tolerance
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
103103103© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Composite SLA for Jitter
• Short answer: NO!
• This is not a valid approach to calculate total jitter based on measured jitter, because we don’t know how to do it…
• Too many factors: positive jitter, negative jitter, percentile-95 of jitter, average jitter,…
• You’d better measure it, not calculate it
2 ms 4 ms 3 ms
Can We Add a Jitter Value to a Jitter Value?
104© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
SA Agent Management Applications
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
105105105© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Management Applications Supporting SA Agent
Internetwork Performance Monitor (IPM)Service Management Suite (SMS)VPN Solution CenterCNS Performance Engine
eHealthVistaViewPowerView
Firehunter
UpTimeIPInsight
Brixworx
…and Many More
106106106© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
SA Agent Applications
TrendReportsTrend
Reports
HealthReports
HealthReports
Report for Thu 1/15/98
Service LevelReports
Service LevelReports
ExceptionsReports
ExceptionsReports
01/15/1998
09/13/1997
09/13/1997
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
107107107© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Screenshot—Infovista Vistaview
108108108© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Screenshot—Cisco IPM Probe Configuration
Target for SA AgentDoes Not Have to Be
a Cisco Router
Device whereSA Agent Is Configured
and Statistics Are Collected Type of Probe andCharacteristics
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
109109109© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Screenshot—Cisco IPM Hourly Jitter
110© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
References(If You Want to Go Deeper)
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
111111111© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
References
• Cisco IOS documentation on CCO:http://www.cisco.com/univercd/cc/td/doc/product/software/ios120/120newft/120t/120t5/saaoper.htm
• SAA User Guide:http://www.cisco.com/warp/public/cc/pd/iosw/prodlit/saaug_ai.htm
• Measuring Delay, Jitter, and Packet Loss with Cisco IOS SAA and RTTMON:
http://www.cisco.com/warp/public/126/saa.html
112112112© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
References
• SA Agent Support for Frame Relay, VoIP, and MPLS VPN Monitoring:
http://www.cisco.com/univercd/cc/td/doc/product/software/ios122/122newft/122t/122t2/ft1csaa.htm
• Deploying Tight-SLA Services on an Internet Backbone (RIPE-41):
http://www.ripe.net/ripe/meetings/archive/ripe-41/eof.html
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
113113113© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Feedback
• Any question, feedback or suggestion regarding SAA, please use the eMail alias:
114114114© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Conclusion/Summary
• SA Agent is a powerful and accurate toolbox providing a comprehensive set of measurement capabilities
• Easy integration into your current network, without additional equipment or skills
• It is free, and available today on almost all platforms running Cisco IOS
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
115115115© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Other Network Management Sessions
• Network ManagementNMS-1001 Introduction to Network ManagementNMS-2001 Network Troubleshooting Tools and Techniques
• FaultNMS-1011 Principles of Fault Management
• ConfigurationNMS-2021 Configuration of Large-Scale Networks with CiscoWorks NMS-4021 Advanced Configuration Methods
• AccountingNMS-1031 Introduction to Collecting Traffic Accounting InformationNMS-4031 Advanced NetFlow Accounting
• PerformanceNMS-1041 Introduction to Performance ManagementNMS-2041 Performance Measurement with Cisco IOS Software
• SecurityNMS-2051 Securely Managing Your Network
• ServicesNMS-1101 Understanding DNS and DHCPNMS-2102 Deploying and Troubleshooting NAT
• High AvailabilityNMS-1201 Improving Network AvailabilityNMS-2201 Deploying Highly Available Enterprise Networks
116116116© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Questions?
116116116© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
117117117© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Please Complete Your Evaluation Form
Session NMS-4041
118118118© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
119© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Annex
120© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
SA Agent vs. IETF IPPM
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
121121121© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
What Is IETF IPPM?
• An IETF working group
• Stands for IP Performance Metrics
• Exists since 1995
• Develop a set of standard metrics that can be applied to the quality, performance, and reliability of IP
• A lot of mathematics and statistics
122122122© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
SAA vs. IPPM
• Is an application
• Has it’s own way to perform measurements
• Address specific protocols and problems
• Is a specification
• Focussed on defining metrics for repeatable and interoperable measurements
• Generic for IP
SAA IPPM
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
123123123© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Some of IPPM’s RFCs
• Framework for IP Performance Metrics (RFC 2330)
• IPPM Metrics for Measuring Connectivity (RFC 2678)
• A One-Way Delay Metric for IPPM (RFC 2679)
• A One-Way Packet Loss Metric for IPPM (RFC 2680)
• A Round-Trip Delay Metric for IPPM (RFC 2681)
• A Framework for Defining Empirical Bulk Transfer Capacity Metrics (RFC 3148)
• One-Way Loss Pattern Sample Metrics (RFC 3357)
124© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
SA Agent in the Future
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
125125125© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
SA Agent—Upcoming Features
• New features coming soon: VoIP active monitoring (Eng2)
Voice over IP network assessment and health monitoring; generate real RTP traffic
VRF backport to Eng1
Port the current VRF-aware SAA features to 12.0(26)S commonly used by SP
Bulk scheduling of SAA probes (Eng2)
Increase scalability when using CLI configuration
Ease of use improvements (Eng2)
Enhanced and more understandable show commands, and a new easy to use MIB
126© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Statistical Considerations
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
127127127© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Mean
• The mean gives some sort of ‘average’
• This is not the median
• It is sensible to noise (spikes)
• To calculate the mean, use:
∑=
=n
kkxn
m1
1
128128128© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Mean (with SAA)
• With SAA output, the mean can be calculated with:
NumOfRTTRTTSum
=m
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
129129129© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Example of Mean
• This example shows two latency samples having the same mean of 120ms; this clearly shows that the mean is not sufficient to determine the behaviour
0
50
100
150
200
1 3 5 7 9 11 13 15 17 19
Sample
Lat
ency
(m
s)
Result 1 Result 2
130130130© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Variance
• The variance is a measure of how spread out a distribution is
• To calculate the variance:
mean. the and samples ofnumber theis Where
)(1 2
1
2
xn
xxn
n
kk∑
=
−=σ
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
131131131© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Standard Deviation
• It is the square root of the variance
• Measures the spread of the data around the mean value
• A smaller standard deviation is better
• Use the formula:
2
1
)(1 ∑
=
−=n
kk xx
nσ
132132132© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Standard Deviation with SAA [1/3]
• When SAA runs jitter probe, about the latency we have only the number of packets, the sum and the sum of squares
• To apply the std dev formula we need the square difference between samples and mean; let’s tweak the formula
• Calculate a corrective factor gamma to add to the sum of squares (RTTSum2), so that:
∑ ∑ +=−n
k
n
kkk xxx γ22)(
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
133133133© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Standard Deviation with SAA [2/3]
• Major steps, the details can easily be done offline (proceed by distribution then simplification):
−=
+−++−=+++=−++−
∑n
kk
n
nn
xxnx
xxxxxx
xxxxxx
2
2...2
...)(...)(22
1
221
221
γ
γγ
134134134© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Standard Deviation with SAA [3/3]
• After due substitution in the standard deviation formula, and simplifications, obtain this formula—that gives the standard deviation for latency based on SA Agent standard output:
• The same approach can be used for one-way latency, one-way jitter and so on
2
NumOfRTTRTTSum
NumOfRTTRTTSum2
−=σ
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr
135135135© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Example
• Result 1 standard deviation is 20,6• Result 2 standard deviation is 2,65
• This is a good indicator
0
50
100
150
200
1 3 5 7 9 11 13 15 17 19
Sample
Lat
ency
(m
s)
Result 1 Result 2
136136136© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2
Applied Example [1/2]
• Here is an execution of a SA Agent Jitter probe, where we will calculate mean and standard deviation:
etychon-vpn#sh rtr op 1Current Operational State
Entry Number: 1[…]RTT Values:NumOfRTT: 10 RTTSum: 193 RTTSum2: 3741Packet Loss Values:PacketLossSD: 3 PacketLossDS: 0PacketOutOfSequence: 0 PacketMIA: 0 PacketLateArrival: 0InternalError: 0 Busies: 0[…]
Top Related