Cover of ACE Analyst Lecturephucchau.tran.free.fr/book/AppTransaction Xpert_Lecture_16.0... ·...

Introduction to ACE Analyst

Lecture Manual

NOTICES

Product Information

Lecture Manual for: ACE Analyst

Software Release: 16.0.T PL0

Technical Support

Telephone: 240.497.1200

Facsimile: 240.497.1064

E-mail: [email protected]

Web: http://www.opnet.com

OPNET Technologies, Inc. (“OPNET”) is the sole owner of all proprietary rights in this Lab Manual (the “Lab Manual”) and the OPNET IT Guru software (the “OPNET Software”).

Terms and Conditions of Use

Access to and use of the Training Manual and/or the OPNET Software is restricted to authorized users who are employees or direct contractors of a Licensee who has accepted the terms and conditions of OPNET’s Software Master License Agreement. As a condition of having access to or use of the Lab Manual and/or the OPNET Software all users agree to be subject the terms of OPNET’s Software Master License Agreement. Users shall not copy or reproduce, in whole or in part, or permit access to or disclose the Lab Manual or the OPNET Software to any other person without the prior written consent of OPNET.

Liability Statement

The terms and conditions governing the licensing of OPNET Software consist of those set forth in the written contracts and license agreements between OPNET and its customers. No representations in this Lab Manual or statements of capability, suitability for use, accuracy or performance, shall be a warranty by OPNET unless contained in a written agreement signed by OPNET.

OPNET DISCLAIMS ALL EXPRESS OR IMPLIED WARRANTIES AS TO THE LAB MANUAL AND THE OPNET SOFTWARE INCLUDING IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.

Destination Control Statement

This Lab Manual and the OPNET Software were manufactured in the United States by OPNET. The initial export of this Lab Manual and the OPNET Software, and any subsequent relocation to another country shall comply with United States law relating to the export of technical data, equipment, software and know-how.

Restricted Rights Legend

USE BY AGENCIES OF THE UNITED STATES GOVERNMENT: For purposes of use of this Lab Manual and OPNET Software by agencies of the United States government (or those authorized to use of this Lab Manual and OPNET Software on behalf of an agency), the OPNET Software is “Commercial Computer Software” within the meaning of DFARs 227.7202 and 252.227-7014(a)(1) and FARs 52.227-19. Use, reproduction and disclosure is subject to DFAR and FAR provisions relating to Commercial Computer Software and the License Agreement between OPNET and the agencies using OPNET Software. OPNET software is provided with a RESTRICTED RIGHTS NOTICE for government use.

Trademarks

OPNET, OPNET Technologies, Inc., OPNET Modeler, IT Guru, OPNET Development Kit, NetBiz and IT Guru are trademarks of OPNET Technologies, Inc. All other trademarks used in the Lab Manual are the property of their respective owners and used herein for identification purposes only.

CONFIDENTIAL INFORMATION DO NOT DISCLOSE, FORWARD, DISTRIBUTE, SHARE, OR MAKE COPIES OF THIS DOCUMENT IN WHOLE OR IN PART. This document contains confidential information and may contain information that is proprietary, privileged, and/or exempt from disclosure under applicable law. This document is intended for the exclusive use of the person to whom it is disclosed. If you are an unauthorized person, you are hereby notified that any viewing, copying, disclosure or distribution of this information may be subject to legal action. All unauthorized persons must immediately destroy the original documentation without making any copies or further unauthorized disclosure.

Introduction to ACE Analyst

Lecture Manual

NOTICES

Product Information

Lecture Manual for: ACE Analyst

Software Release: 16.0.T PL0

Technical Support

Telephone: 240.497.1200

Facsimile: 240.497.1064

E-mail: [email protected]

Web: http://www.opnet.com

OPNET Technologies, Inc. (“OPNET”) is the sole owner of all proprietary rights in this Lab Manual (the “Lab Manual”) and the OPNET IT Guru software (the “OPNET Software”).

Terms and Conditions of Use

Access to and use of the Training Manual and/or the OPNET Software is restricted to authorized users who are employees or direct contractors of a Licensee who has accepted the terms and conditions of OPNET’s Software Master License Agreement. As a condition of having access to or use of the Lab Manual and/or the OPNET Software all users agree to be subject the terms of OPNET’s Software Master License Agreement. Users shall not copy or reproduce, in whole or in part, or permit access to or disclose the Lab Manual or the OPNET Software to any other person without the prior written consent of OPNET.

Liability Statement

The terms and conditions governing the licensing of OPNET Software consist of those set forth in the written contracts and license agreements between OPNET and its customers. No representations in this Lab Manual or statements of capability, suitability for use, accuracy or performance, shall be a warranty by OPNET unless contained in a written agreement signed by OPNET.

OPNET DISCLAIMS ALL EXPRESS OR IMPLIED WARRANTIES AS TO THE LAB MANUAL AND THE OPNET SOFTWARE INCLUDING IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.

Destination Control Statement

This Lab Manual and the OPNET Software were manufactured in the United States by OPNET. The initial export of this Lab Manual and the OPNET Software, and any subsequent relocation to another country shall comply with United States law relating to the export of technical data, equipment, software and know-how.

Restricted Rights Legend

USE BY AGENCIES OF THE UNITED STATES GOVERNMENT: For purposes of use of this Lab Manual and OPNET Software by agencies of the United States government (or those authorized to use of this Lab Manual and OPNET Software on behalf of an agency), the OPNET Software is “Commercial Computer Software” within the meaning of DFARs 227.7202 and 252.227-7014(a)(1) and FARs 52.227-19. Use, reproduction and disclosure is subject to DFAR and FAR provisions relating to Commercial Computer Software and the License Agreement between OPNET and the agencies using OPNET Software. OPNET software is provided with a RESTRICTED RIGHTS NOTICE for government use.

Trademarks

OPNET, OPNET Technologies, Inc., OPNET Modeler, IT Guru, OPNET Development Kit, NetBiz and IT Guru are trademarks of OPNET Technologies, Inc. All other trademarks used in the Lab Manual are the property of their respective owners and used herein for identification purposes only.

CONFIDENTIAL INFORMATION DO NOT DISCLOSE, FORWARD, DISTRIBUTE, SHARE, OR MAKE COPIES OF THIS DOCUMENT IN WHOLE OR IN PART. This document contains confidential information and may contain information that is proprietary, privileged, and/or exempt from disclosure under applicable law. This document is intended for the exclusive use of the person to whom it is disclosed. If you are an unauthorized person, you are hereby notified that any viewing, copying, disclosure or distribution of this information may be subject to legal action. All unauthorized persons must immediately destroy the original documentation without making any copies or further unauthorized disclosure.

2/10/2011

1

OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc.

Introduction to AppTransaction Xpert (ACE Analyst)

test


Day 1

Corporate Overview• Founded in 1986• Publicly traded (NASDAQ: OPNT)• HQ in Bethesda, MD• Approximately 600 employees• Worldwide presence through direct offices and channel

About OPNET Technologies, Inc.®

partners• Global distribution agreement with Cisco

Best-in-Class Solutions and Services• Application Performance Management• Network Performance Management• Network R&D

Strong Financial Track Record• Long history of profitability

test

g y p y• Trailing 12-month revenue of over $100M• Approximately 25% of revenue re-invested in R&D

Broad Customer Base• Corporate Enterprises• Government Agencies/DoD• Service Providers• Network Equipment Manufacturers

3 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc.

2/10/2011

2


Customers (partial list)

Energy / UtilitiesGovernmentFinancial Services Insurance

Healthcare / Bio / Pharmaceutical Manufacturing Media / Communications Services

test

Retail / Consumer Products Transportation / Logistics

OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc.4 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc.

OPNET’s APM Xpert Suite

End-user experience and network monitoring and analytics

Deep application component monitoring and analytics

Deep transaction tracing, analysis, and prediction

Passive database performance monitoring

test

5 OPNET Confidential – Not for release to third parties

© 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc.

Network and server discovery and monitoring

“What's in your Application?”

2/10/2011

3


OPNET is an IACET Authorized Provider

OPNET Technologies, Inc. has been approved as an Authorized Provider by the International Association for Continuing Education and Training (IACET), 1760 Old Meadow Road, Suite 500, McLean, VA 22102. In obtaining this approval, OPNET has demonstrated that it complies with the ANSI/IACET 1-2007 Standard which ishas demonstrated that it complies with the ANSI/IACET 1 2007 Standard which is internationally recognized as the Standard for training organizations.

As a result of their Authorized Provider status OPNET is authorized to offer IACET Continuing Education Units (CEUs) for its programs that qualify under the ANSI/IACET 1-2007 Standard.

test


Continuing Education Units

What is IACET?

IACET is a Standards Developing Organization accredited by the American Standards Institute (ANSI) and is an authorizing body for education providers that meet the American National Standard for continuing education and training (ANSI/IACET 1-2007 Standard). The IACET standard is what learners seek for quality and is the core of thousands of educational programs worldwide. (see: www.IACET.org)

What is an Authorized Provider?

The IACET Authorized Provider (AP) status means that an organization adheres to IACET’s internationally recognized ANSI/IACET 1-2007 Standard for Continuing Education and Training. Only Authorized Providers may award IACET Continuing Education Units (CEUs).

Wh t i CEU?test

What is a CEU?

The Continuing Education Unit (CEU) was created by IACET as a measurement of continuing education. One (1) IACET CEU is equal to ten (10) contact hours of participation in an organized continuing education experience under responsible sponsorship, capable direction, and qualified instruction.


2/10/2011

4


Earning CEUs for this Course

Introduction to AppTransaction Xpert – This course is worth 1.3 CEUs

• To receive these CEUs, you must– Attend the entire course

– Fill out the feedback form

– Score 80% in a test at the end of class

• Your test is scored in the next 10 business days

• If successful, you will receive a certificate denoting your CEUs in the mail within 2 weeks

test

within 2 weeks

• Your completion of the course is stored in our database for later retrieval as needed


Let’s get to know each other…

• Name ?

• What company you work for ?

• Which Department ?

• What experience do you have with OPNET products?

• Did you know we changed the names… !!??

test


2/10/2011

5


Course Structure

Day 1

Introduction to AppTransaction Xpert

Day 2

Advanced AppTransaction Xpert

•About OPNET

•Visualizations in AppTransaction Xpert

•Troubleshooting application performance

•Predicting application behavior before deployment

•Advanced Topics

•Understanding Delay Components

•Parallel Effects, User think Time

•Troubleshooting TCP applications

test

deployment

•Capture and import strategies

•WAN Acceleration


Class Format

• In general, four phase instruction

– Short lecture sections

– Demonstrations

– Related hands-on labs

– Discussion of lab applications

• Class is interactive – don’t hesitate to ask questions

• Please fill out the feedback form online @ eval.opnet.com

test– Give us information on your current network and application concerns

– Provide suggestions on how we can improve our products

– Let us know how we can improve this training class


2/10/2011

6


Agenda Introduction

- Licensing Options

- Overview, Use Cases

- Basic Workflow

Using AppTransaction Xpert to visualize applications- AppTransaction Xpert Visualizations

- Lab 1: Visualizing Application

Using AppTransaction Xpert to diagnose problematic applications- AppDoctor

Reporting capabilities of AppTransaction Xpert

R ttest

- Reports- Lab 2: Application Performance Troubleshooting

Using AppTransaction Xpert to perform performance predictive studies - QuickPredict and QuickRecode

- Lab 3: Predictive Analysis


Licensing Options

• Download and administer licenses using the License Manager

• Licensed on concurrent user basis

• Online license management g(http://www.opnet.com/support/license_management.html)

• Licensing options

– Standalone vs. Floating (license server)

– Loanable Licenses

test


2/10/2011

7


Overview of AppTransaction Xpert

test


AppTransaction Xpert Solution

• AppTransaction Xpert Plus

• AppTransaction Xpert Standard

• IT Guru with the following modules

– ACE

– ACE Decode Module

– Advanced ACE Capabilities

• Starting with version 16.0, the different offerings of AppTransaction Xpert (that is: AppTransaction Xpert Plus, AppTransaction Xpert Standard, and AppTransaction Xpert Module) have been combined into a single installer.

test

Important—Existing users are not losing any AppTransaction Xpert functionality

because of this change. Functionality is determined by licenses.


2/10/2011

8


What is AppTransaction Xpert... ?• Application Characterization Environment

• Capture application traces

Vi li A li ti D i• Visualize Application Dynamics– Application layer analysis– Network layer analysis

• Diagnose performance problems– Minimize troubleshooting time

test• Validate solutions

– Predict application behavior under different network conditions– Evaluate performance impact of application design decisions– Validate infrastructure readiness for new deployments

• Generate reports16 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc.

When to use AppTransaction Xpert ??

(Use Cases)test


2/10/2011

9


Use Case 1

Troubleshooting application performance –

• Users of a critical application are reporting very bad response times

– Network engineers are reporting no issues on the network

– Application engineers are reporting no issues with software

• Result: Finger pointing !!

Wasted time

test


Use Case 2

Testing changes of the enterprise environment –

• Relocation of key processing data center from US to China

• Management needs to know the implications of move

– Impact on application performance and network utilization?

– Will additional resources be required?

• Result: Need to produce accurate performance projections and network l i i

testplanning requirements


2/10/2011

10


Use Case 3

Application pre-deployment analysis –

• Purchase of a new 3-party application system

• Management needs to know how the new system will perform

– Application response times

– Network utilization

• Result: Need to eliminate the need for trial and error deployment of

testapplication. Expensive and time consuming.


Basic Workflow

test


2/10/2011

11


What does AppTransaction Xpert tell you … ?

• “Why is my application slow?”

– User’s desktop running slow?

– Server delays?y

– Non-optimal protocol settings (TCP and others)?

– Heavy congestion over WAN?

– Too little bandwidth?

– Application too chatty?

• “Can this new application be deployed over our network?”

test– Expected transaction response times?

– Impact other critical applications?

– Bandwidth requirements?


How does AppTransaction Xpert tell you this...?

• Reports, graphs, and tables

• Visualization of application behaviorpp

• Diagnosis of bottlenecks and performance problems

• Predictive analysis of same traffic on different network

• Input to other OPNET products for further analysis

test– Application traffic analysis

– Capacity planning studies


2/10/2011

12


Inputs to AppTransaction Xpert

• Packet traces from the following sources– ACE Capture Agents

ACE Live– ACE Live– Cisco NAM Blade– WAN Accelerator– F5 BIG-IP– Sniffer– UNIX (no agent installed)– Tcpdump, windump ..etc

test• Basic characteristics of network (Recommended)

– Tier locations– Bandwidth between tiers– Latency between tiers


Application Troubleshooting Workflow

Identify performance issues

Capture Packet Traces

Optional: Create IT Guru

Visualize, Diagnose, and Predict

Optional: Detailedtest

Optional: Create IT Guru network model

Generate Reports

Optional: Detailed Analysis


2/10/2011

13


AppTransaction Xpert Visualizations

test


AppTransaction Xpert Visualizations

• Data Exchange Chart– Shows timeline of data sent

• Tier Pair CircleTier Pair Circle– Shows summary statistics

• Tree View– Shows Transaction Analysis

• Decodes– Shows Application layer headers

test• Graphs– View time-based graphs of application statistics


2/10/2011

14


Data Exchange Chart (DEC)

• Timeline of application messages and network packets between tiers

• Slope indicates time to cross network

• Color indicates payload size of message

test


DEC: Message Groups

• When messages and packet are too frequent to be displayed individually, a solid block is shown

• Block is a histogram showing the size of messages in groups

• Zoom in or use tooltip to see more detail

test


2/10/2011

15


DEC: View Options•Application Chart Only

Shows every application message – several packets may comprise a message

• Network Chart Only

Shows every individual packet in the trace file, including TCP acknowledgements

• Application and Network Charts

Shows both charts in the same window

testApplication Messages

Network Packets


DEC: Show Dependencies

• Triangles appear at largestchunks of delay

Click “Show Dependencies”

• White = delay at host

• Orange = user think time

• Purple = network delay

test

LegendLegend


2/10/2011

16


DEC: Application Characteristics

• Quiet Time

– User think time?

– Slow client computer?p

– Slow server?

– Not seeing the whole picture?

§Tip: See what happened before and after silence

test

f

Tip: Check Network Chart for more details

Large periods of silence


DEC: Application Characteristics (cont.)

• Chatty Application

– Lots of back-and-forth communication

(Application Turns)

Lots of small messages (orange and yellow)

test

Groups of messages close together


2/10/2011

17


Application Turn

• Counted on the Application Layer (Application Messages)

• Every turn has to cross the network latency which adds up to the total Latency Delay

2

Client

Networktest

1 3

Server

NetworkLatency


DEC: Application Characteristics

• One-way data transfers– Dark blue arrows

Most arrows in same direction– Most arrows in same direction

Tip: Check the Network Packet Chart for more details

test

Zoom to see the arrow heads


2/10/2011

18


DEC: Application Characteristics (cont.)

• Lots of irrelevant traffic– Do not try diagnose

application untilapplication until you removed irrelevant data

test

Tip: Go to Tier Pair Circle and delete unrelated traffic


Tier Pair Circle

• Summary of traffic between all tiers

– A high-level view of theg

application task

– Is there extraneous traffic?• Right-click to exclude

unwanted traffic

test

Tip: If you don’t see the protocol you expect, then right-click to re-recode traffic on the fly


2/10/2011

19


Tier Pair Circle: Functionality

• Choose statistics to view

• Right-click a tier or conversation to exclude /delete it or all others

• Show only traffic containing a specific protocol

• Find Top Statistics to sort test

pconversations by relevance


Tier Pair Circle: Statistics

• Pull down to select statistics to view:

test


2/10/2011

20


Tree View

• Tree pane: One-line summary of tier pairs, connections and frames

• Table pane: Detail statistics by tier pairs, connections and frames

• Timeline pane: Overview of messages over the time of the taskp g

test

Tree Pane Table Pane Timeline Pane


Tree View: “View by:” Options

• Tier Pairs – Network Packets

• Tier Pairs – Application Transactions

• Application Transactions

Group by:

test


2/10/2011

21


Tree View: Sorting

• Click a column header to sort the table by descending values.

• Click the table header again to sort by ascending values.

test


Tree View : Customize the Table Pane

• Choose View > Choose Table Columns…

• Right-click on the column header and click “Choose TreeView Columns…”

• Select the network statistics to appear in the table

test


2/10/2011

22


Decode Viewer

ETH IP TCP HTTP ETH

• What are decodes?

– Analysis or description of the bytes at each layer of a packet

test• What can you learn from decodes?Detailed protocol information

Information about data being transferred

Ex: TCP gives Sequence number and Acknowledgement number


Embedded Decode Viewer

• Show / Hide the embedded decode viewer by clicking and

• Select traffic in the top pane and see corresponding decodes in the bottom pane

• Right-click on selected

testframes to export to spreadsheet

• Click the plus sign to expand layers of decodes


2/10/2011

23


Tree View to DEC

• Display Context-Based Traffic in Data Exchange Chart —

– Double-click on traffic for a tier pair in the Timeline pane to display the corresponding context- based traffic in the Data Exchange Chart.

test


Graphs

• Time varying statistics

– Advanced diagnosis of network and TCP problems

test


2/10/2011

24


Graphs (cont.)

• Select graphs to display

• Options:

– Specify bucket width for precision of graph

– Embed graphs in DEC for easy comparison against messages in DEC

test– Overlay similar graphs or overlay all

• Select “Show”


Graphs: Bucket Width

• What is a bucket width?

– Statistics can be averaged over time rather than shown as individual data points

– A bucket is the duration of data you want to average, like 100 ms or 3 seconds

• Same trace, same statistics, different bucket widths:

test

d e e t buc et w dt s:

– 100 ms

– 3000 ms (3 seconds)


2/10/2011

25


Graphs: Embedded Statistics

• Time-aligned statistics with the Data Exchange Chart

test


LAB 1

LAB 1

Visualizing

Application

test


2/10/2011

26


Lab 1: Visualizing Application

• Overview

– The ACE trace file represents a three-tier Oracle application. You will p ppexplore some of the different visualizations in AppTransaction Xpert that you can use to troubleshoot and analyze the application.

• Starts on page 1 in your lab manual

test


Lab 1: Conclusion

AppTransaction Xpert provides many different views of a transactionto understand the interaction of the application on the network

• Data Exchange ChartData Exchange Chart– Shows timeline of data sent

• Tier Pair Circle– Shows who talks to whom and summary statistics

• Tree View– View the network packets in order, with a decode summary– Transaction analysis

• See the timeline of transactions• Decodes

test

Decodes– Drill down into TCP, IP, or even application layer headers to understand

application behavior• Graphs

– View time-based graphs of throughput, congestion, and other statistics


2/10/2011

27


AppDoctor

test


What is AppDoctor … ?

• Invoke AppDoctor– AppDoctor (menu)– AppDoctor (icon)

• AppDoctor Tabs

– Summary of Delays

– Executive Summary

– Diagnosis

– Statistics

– Protocol-specific

test


2/10/2011

28


AppDoctor: Summary of Delays

• Pie chart that breaks down the total response time into processing, network, and

ll l ff tparallel effects

• Most useful view in AppTransaction Xpert when troubleshooting applications

• Quickly points you to what is

testthe problem, or at least what is definitely not the problem

• Must be used on a single application task


Summary of Delays: Processing Effects

• Tier Processing delay

– Time spent processing on each tier

test


2/10/2011

29


Summary of Delays: Network Effects

• Latency– Delay caused by the physical

distance between tiers– Common problem in chatty p y

applications• Bandwidth

– Delay caused by transmission (i.e. putting the bits onto the wire)

• Protocol – Could be non-optimal TCP

settings – Check Diagnosis tab for

test

Check Diagnosis tab for bottlenecks

– Check Statistics tab for values• Congestion

– Delay caused by other traffic in the network


Summary of Delays: Parallel Effects• Delay incurred when multiple types of delay (such as tier processing,

bandwidth, and latency) occur simultaneously

test


2/10/2011

30


AppDoctor – After Network Definition (16.0 )

test

BEFORE AFTER


Summary of Delays: Drill-Downs

• Double-click any section of the pie chart for extra information

• Drill-downs give more information that can help diagnose problems

• Ex: Bandwidth drill-down shows throughput graphs and link bandwidthg p g p

test


2/10/2011

31


AppDoctor: Executive Summary

• Gives overview information for application task– Total transaction time– Total application data transferred

Si l t t f d l– Single greatest cause of delay– Tiers in application– Bandwidth and latency at each tier

test


AppDoctor: Diagnosis• Displays common

application, network, and server problems

• Flags bottlenecks and• Flags bottlenecks and potential bottlenecks

• Various characteristics of the ACE file are compared with thresholds

• Click on any cell

test

Click on any cell (especially on bottlenecks) for extra information


2/10/2011

32


AppDoctor: Statistics

• Tabular view of key application characteristics

• “Effect of” statistics quantify• Effect of statistics quantify the components of delay

• Other statistics include– Application message statistics

– Network packet statistics

– TCP statistics

testTip: Use it with Diagnosis tab to quantify bottlenecks

Tip: You can export to spreadsheet


AppDoctor: Protocol-Specific Tabs

• Protocol-specific information, statistics, and diagnoses

• Appear automatically when certain protocols are detected in an ACE file:

– HTTP

– Citrix

– CORBA

– SOAP (Web Services)test

SOAP (Web Services)


2/10/2011

33


Right-Click Operations in AppTransaction Xpert

• Graph Statistics such as throughput or congestion

• Exclude or permanently delete packets and messages

• Include packets and messagesp g

• Rename a tier or lookup a tier name with DNS

• Re-code traffic

• Zoom in and out in the DEC

• Show protocol decodes for selected traffic

• Export decodes for selected frames in the decode viewer

test


Take-Away Points

AppTransaction Xpert provides an analysis of the main application bottlenecks that is easy to use and to understand. The diagnosis is displayed and broken down into components of delay. The different components are:

• Processing Effects

• Network Effects– Latency– Bandwidth– Protocol (with recommended options)

C titest

– Congestion

• Other Effects– Parallel Effects: Combination of any processing effects and network

effects that occur at the same time


2/10/2011

34


Reports

test


Reporting Motivation

• Quick and easy to use

• Support your findings

• Recording and dissemination of all analysis results to all interested stake holders

• Documentation of application behavior for future reference

test


2/10/2011

35


Report Selection - Type and Format

• Very simple and quick – select report type for output (from pull down menu, or icon button)

test


Report Options

• Select report options and generate report– Report sections, title (statistics)– Report directory (output), Report Server (optional)

test


2/10/2011

36


ACE – MS Word Report

test


ACE – Web Report URL

Home Dir(all reports)

test


2/10/2011

37


ACE – Multiple Transaction Report

Trans 1

Trans 2

test


ACE – Difference Report

test


2/10/2011

38


ACE – Spreadsheet Export Reports

Comparison Report

Tree View Statistics

AppDoctor Statistics

test


• Report template — Defines the previously-defined templates used

to generate the report. You can modify and customize report templates.

The standard templates:

ACE – Report Template

– ApplicationDeveloperReport.ace.rt.txt

– ManagementReport.ace.rt.txt

– NetworkEngineerReport.ace.rt.txt

– concise_report.ace.rt.txt

– detailed_report_ace.rt.txt

– standard_report_ace.rt.txt

• Reports can be cataloged and managed by the Report Servertest

• Reports can be cataloged and managed by the Report Server– Provides security, allows access by only registered users

– Provides automatic ageing of reports


2/10/2011

39


LAB 2

LAB 2

Application

Performance

Troubleshootingtest


Lab 2: Application Performance Troubleshooting

• Overview– Response time problems for remote users of new web application

– Remote site connected via a 256-Kbps Frame Relay circuit

– Need to determine cause of the delay (network or web server)

– Traces have been captured at both server and client

• Objectives– Visualize application in ACE

– Identify the cause of the application delay

– Use Transaction Analysis to identify problem with application

test• Starts on page 9 in your lab manual


2/10/2011

40


Lab 2: Conclusion

• Conclusions

– The web page takes approximately 24 seconds to download

– Bandwidth and congestion are the biggest sources of delay

– Server delay is not a problem

– Majority of traffic flow is from the server to the client

– Web page consumes almost all of the 256-Kbps circuit bandwidth

– Largest transaction is a download of a 352 KB graphic

test– This single graphic takes 18 seconds to download


Performance Predictive Studies

The use cases for predictive studies:

• Predict application behavior under different network conditionspp– Problematic applications

• Evaluate performance impact of application design decisions – Modified applications– Problematic application

• Validate infrastructure readiness for new deploymentstest

Validate infrastructure readiness for new deployments


2/10/2011

41


Prediction in AppTransaction Xpert

• What can predictive analysis tell you?

– Response time of application under different circumstances

– Will adding bandwidth help?g p

– Is this application sensitive to latency?

– Can we roll this out to remote users?

• What sort of studies will it help with?

– Troubleshooting performance problems

– Testing a new application

test– Deploying a current application to new users

– Server relocation studies


Application Deployment Workflow

Identify performance issues

Capture and import into ACE

Create report and prediction scenario templates

Use QuickPredict to predict performance

O ti l U M lti Utest

Optional: Use Multi-User QuickPredict for basic capacity

planning

Optional: Use IT Guru for network and server capacity

planning83 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc.

2/10/2011

42


Application Deployment Workflow

• Optional: Use simulation for further studies

– Discrete Event Simulation to predict response times or perform server capacity planning

– Flow Analysis for network capacity planning

– Simulation covered later in course

test


QuickPredict

test


2/10/2011

43


QuickPredict

• Simplified simulation engine

• Helps study effects of the network on the application

• Questions QuickPredict can answerQ Q

– How much bandwidth would it take to get a 4 second response time?

– Would increasing the TCP Window improve response time?

– What happens to response time if latency increases?

– How sensitive is this application to packet loss?

– What would the response time be if the link was 65% utilized by other traffic?

h if h d 10% A A l i i k !!test

– What if I had a 10% WAN Acceleration in my Network – NEW !!


QuickPredict: Outputs• Two different output styles

– QuickPredict Bar Charts shows components of delay for different scenarios

– QuickPredict plots response time over range of network conditions

test


2/10/2011

44


QuickPredict Bar Charts: Scenarios

• Click to add multiple scenarios

• Right-click to remove scenarios

test


QuickPredict Bar Charts: Options

• Put parameters on slider

– Drag slider back and forth to dynamically update bar chart

• Rename scenarios

• Symmetric Bandwidth

– Set asymmetric values for bandwidth

• Label delay categories

• Load and save parameter settings using templates

test

using templates


2/10/2011

45


QuickPredict Bar Charts: Options – NEW

Introduce WAN Acceleration• Byte Reduction • Turn Reduction

C i• Compression • Caching

test


QuickPredict Bar Charts: Tabular Results

• View detailed results for each scenario in the Tabular Results tab

test


2/10/2011

46


QuickPredict: Response Time Graphs

• Shows sensitivity of task response time to network characteristics:

– Bandwidth

– Latencyy

– Packet Loss

– Link Utilization

– TCP Window Size

– WAN Acceleration

test


QuickPredict: Advanced View

• Choose network path to modify

• Choose an X-axis parameter and the range of values over which to plot response times

• Select values for each of the other parameters

• Click Update Graph

• Overlay multiple curves on same graph

• Load and save templates

test


2/10/2011

47


QuickPredict: Table View

• Modify parameters on all network paths in the task

–Red text indicates X-axis parameter–Use Set X-axis to specify X-axis parameterp y p–Load and save templates

test


QuickRecode

test


2/10/2011

48


QuickRecode

• Changes high level application characteristics– Shows sensitivity of task response time to application characteristics:

•Application turns•Total application bytes sent•Total processing time in tiers

– Use with QuickPredict to see how response times are affected

test


QuickRecode: Use Cases

• Use Cases– Performance benefits– Changing database access

to reduce chattiness– Demonstrate effect of faster

app serverHint: Use QuickRecode to

prove effect of optimizing application

test


2/10/2011

49


LAB 3

LAB 3

Predictive

Analysis

test


Lab 3: Predictive Analysis

• Overview– Deploying a new application to several remote locations– Application has client querying an Oracle database– Application has been deployed to pilot site (in same city as database) and

poor performance is already apparent– Goal is to determine if the application deployment will be successful– If not, what can be done to improve performance?

• Objectives– Perform a deployment study using QuickPredict Bar Charts– Use AppTransaction Xpert to drill down into the app and see what’s going

teston

– Use QuickRecode to test a possible fix



2/10/2011

50


Lab 3: Conclusion

• Always try to determine “goal” or “objective” of any problematic application– Is there an SLA expectation– What is the user expectations– What are other users, departments, experiencing

• Additional useful troubleshooting information to determine– Is the problem reproducible or is it transient– Are all transactions of the application behaving the same

test


Agenda Capture Methodologies

Capture Data Sources

- New Capture Manager Options

Using AppTransaction Xpert to perform packet capturesg pp p p p p- Capture Types

- Lab 4: Capture Process

Using AppTransaction Xpert to filter packet captures

- Capture Filters

Using AppTransaction Xpert to import packet captures- Import Process

L b 5 Id tif i P f P bl i U filt d Ttest

- Lab 5: Identifying Performance Problems in Unfiltered Traces


2/10/2011

51


Capture Methodologies

test


Basic Capture Methodology

• Define and understand the problem at hand– What is slow?– Is it the login or another specific task?– How often does it occur?

• What is a task?– Isolate an “atomic” Transaction– Loading single web page– From when user clicks “Submit” to screen refresh

• Plan your capture– Simultaneous or single-sided– Which tiers are involved in the application?

test• Aim for as “clean” a packet trace as possible– Keep measurement as simple as possible– Filter out unrelated traffic– Capture one occurrence of one task at a time– Perform capture numerous times to help eliminate anomalies


2/10/2011

52


Atomic Transaction Example (Task)

• Web Application of a Bank– Load Login Page

• Open a Web Browser• Type URL of the Bank yp• “Enter”• Login Page Loads

– Login• Type User credentials • “Login”• Home Page shows up

– Check Savings Account• Mouse/cursor around• Click on “Savings Account”

Red arrow = Start Capture

Blue arrow = Stop Capture

test

Click on Savings Account• Savings Accounts Page Loads

– Logout• Call up Logout screen• “Submit”• Logged out page



test


2/10/2011

53



• Packet traces are the primary data inputs

• AppTransaction Xpert supports a wide variety of packet sourcespp p pp y p

– ACE Capture Agents

– ACE Live

– ACE Enterprise Management Server

– Agentless Capture Options – NEW!

• F5 Load Balancers

• UNIX, Linux platforms

test• WAN Accelerators

• Cisco NAM

– Wireshark and other Winpcap and Libpcap sources

• TCPDump, Snoop


Agentless Capture Options – NEW

• No software installation required

• Use the AppTransaction Xpert Capture Manger to control captures remotely via SSH

• Supported on platforms that support TCPDump– TCPDump file remains on machine until capture is completed, then deleted after

retrieval

• Included in the AppTransaction Xpert Plus license

test


2/10/2011

54


Agentless Capture Options – NEW (cont.)

• F5 Load Balancers– Excellent way to follow a transaction to the proper server– Perform successful trace file merging across firewalls

Rule out the device as a source of delay– Rule out the device as a source of delay

• UNIX-Based Operating Systems– Supported Platforms

• Linux• Solaris SPARC• Solaris Intel/AMD• AIX

test• FreeBSD• HP-UX Itanium• Macintosh PowerPC• Macintosh Intel• True64


Agentless Capture Options – NEW (cont.)

• WAN Acceleration Devices

– Support for Riverbed and Cisco

– Restore visibility of the WAN

– More information can be found in session 1471 - Monitoring and Troubleshooting Applications in a WAN Accelerated Environment with ACE Live™ and AppTransaction Xpert

test


2/10/2011

55


Capture Data Sources – AppResponse Xpert (ACE Live)

• New seamless 1-click integration– Let ACE Live do the filtering for you

• Easily pull traces from alert snapshots f t l li ti iof actual application issues

test


Capture Data Sources – AppForensics Xpert (ACE EMS)• ACE Enterprise Management Server

– Provides web-based extension to Continuous Capture functionality– Supports both End-User Driven Capture workflows, as well as Server-Driven

Capture WorkflowsCapture Workflows

test

– End Users report application slowness• Trouble ticket is created with trace files of the incident attached• Troubleshooters claim problem reports and troubleshoot on their schedule


2/10/2011

56


AppForensics Xpert – Web Capture Retrieval

test


Capture Data Sources – AppForensics Xpert

• Server-Driven Capture Workflow– Panorama provides forensic actions based on threshold violations within the

application servers

– The trace file forensic action automatically generates a trace that is accessedThe trace file forensic action automatically generates a trace that is accessed through AppForensics Xpert

– For more detail, attend session 1482 – Deploying an Enterprise-Wide Packet Capture Architecture with ACE EMS and ACE Live

test


2/10/2011

57


Capture Data Sources – ACE Agents

• Benefits of using ACE Agents– Unlicensed– Support for major platforms (Windows, HP-UX, AIX, Solaris, Linux, Mac OS X,

FreeBSD)FreeBSD)– Lightweight, Secure, and Stable– Silent Install capability– Server Statistics collection (Windows)– Enhanced Filtering (NEW) – On-Demand and Continuous Capture modes

test


Additional Data Sources

• AppTransaction Xpert can import a variety of additional data– Identify these sources during capture to ensure collection

• From OPNET products– Windows ACE Capture Agents can collect Lightweight Server Statistics

– OPNET Panorama

• Server statistics and performance metrics

• Java transaction tracing

• Third-party sources– HP GlancePlus

– Perfmon

test– Custom data in commasaved (.csv) format


2/10/2011

58


Capture Lightweight Server Statistics

• Capture Windows performance data – ACE Capture Agent generates two files

– Application traffic (*.appcapture) file

– Windows performance data (*.appcapture.da) file

Display internal server metrics • Percent Processor Time•File Read Bytes Per Second•System Calls Per Second•Available MB

test

Available MB•File Write Bytes Per Second


AppInternals Xpert (Panorama) Server Statistics

• Capture Server Statistics from AppInternals Xpert – Use with on-demand captures

– Thousands of available metrics

• For more information:– Session 1461- OPNET Panorama® Essentials: Multi-Tier Application Monitoring,

Correlation, and Troubleshooting

test


2/10/2011

59


AppInternals Xpert Transaction Trace

• AppInternals Xpert can create a trace of all activity through the Java Virtual Machine (JVM) or .NET CLR– Provides a call-by-call sequencing (or call stack) of all activity within your

li iapplication

• This transaction trace can be imported into AppTransaction Xpert to explain what is occurring on a given system during long processing delays

test


Capture Manager

• Capture Manager control parameters

test


2/10/2011

60


Capture Agent Settings

test


Capture Types

test


2/10/2011

61


On-Demand Capture

•Methodology–User knows exactly when a problem is going to occur and

what it takes to reproduce itwhat it takes to reproduce it

–Deploy and configure ACE Capture Agents

–Start the capture

–User re-creates the problem

–Stop the capture

–Import capture for analysistest

Import capture for analysis


On-Demand Capture (cont.)

test


2/10/2011

62


On Demand Capture Process: Adding an Agent

test


On Demand Capture Process: Specify Agent Settings

test


2/10/2011

63


Capture Process: Start Capture

test


Capture Process: Finish Capture

test


2/10/2011

64


Capture Process: Open in Transaction Analyzer

test


Scheduled Capture

•Useful for reproducible problems during off hours

•Methodology–User knows when a problem will occur

• For example, after-hours batch process or database backup

–Schedule a capture• Choose one-time or repeating capture

• Set the schedule for a capture

• Specify schedule for download

I t t f l itest

–Import trace for analysis


2/10/2011

65


Scheduled Capture (cont.)

test


Scheduled Capture (cont.)

Scheduled capture can create one-time, daily, weekly, or monthly capture schedules

test

Scheduled capture utility can optionally create a one-time, daily, weekly, or monthly schedule to download trace files


2/10/2011

66


Continuous Capture

•Useful for troubleshooting intermittent problems

•Methodology–User experiences intermittent problem

–Start continuous capture

–User calls help desk• Problem occurred again around 9:30

–Preview capture

Download trace of problematic time spantest

–Download trace of problematic time span

–Import trace for analysis

•Agent details–Same capture agents as on-demand capture


Continuous Capture (cont.)

test


2/10/2011

67


Continuous Capture: Preview/Download

test


LAB 4

LAB 4

Capture

Process

test


2/10/2011

68


Lab 4: Capture Process

• Overview

– Configure a capture agent

– Perform an on-demand capturep

– Open in Transaction Analyzer

• Skills covered

– Learn how to configure a capture agent

– Understand how to use agent filters

– Capturing from the network

test• Starts on page 32 in your lab manual


Lab 4: Conclusion

• Configured capture agent for capture– Located and specified host IP Address

– Specified capture mode

– Set up a filter

• Performed a capture– Started the agent

– Performed a transaction

– Stopped the agent

– Brought the capture file into Transaction Analyzer

test

g p y


2/10/2011

69


Capture Filters

test


Filtering Unrelated Traffic

• You want to capture the transaction of interest– AppTransaction Xpert lets you filter out irrelevant traffic

• Options to filter during capture, during the import or after the import– The earlier the better

• Why filter during capture?– Only capture the specific transaction– Smaller trace file to process => easier/faster import

• Why not filter during capture?– Sometimes you want to see the whole picture– If you’re not 100% sure about application path

test• Recommended methodology– Capture without filters for short duration– Examine trace– Determine best filter to use– Capture subsequent tasks with filter


2/10/2011

70


Filter: During Capture

Users can filter out or only allow traffic based on a specific protocol

Users can filter specific hosts and/or ports for incoming or outgoing traffic

New Filtering Options intest

Users can elect to exclude or allow broadcast and multicast traffic

New Filtering Options in Capture Agent 3.9 Use wildcards and slash notation Use DNS names Port ranges Broadcast and multicast can be

excluded independently140 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc.

Filter: During Capture (cont.)

• Preview in Continuous Capture– Select the IP addresses, conversations and time ranges of interest

• Great for troubleshooting intermittent issues

test


2/10/2011

71


Filter: During Import

•What is Trace Explorer?– Graphical protocol analyzer

– Used to help filter unrelated traffic pduring import process

•Why would you use it?– Graphically filter a capture file

– Allows for fast read of large trace files

– You want to know what you’re looking at

test

looking at


Filter: After Import

• Once capture file has been imported, users can still filter irrelevant traffic

– Right-click on any tier pair conversation, application message,

li i i kapplication transaction or network packet

• Option to Exclude or Permanently Delete

– Filter Wizard• Remove unidirectional, non-IP, UDP,

broadcast traffic

– Capture File Filter

test• Use to remove traffic by host, port,

protocol


2/10/2011

72


Import Process

test


Import Overview

• Now that the data has been captured, it must be imported into AppTransaction Xpert

• Review what has been captured– Do you have a single capture file?

– Do you have more than one capture file?

– Are you capturing WAN accelerated traffic?

– Is the capture file filtered?

• If not, use Trace Explorer

test• Greatly simplified for 16.0 !!


2/10/2011

73


Import Changes in 16.0

• Import process has been greatly simplified– 15.0 required answering a series of questions about tier locations, bandwidths, and

latencies

– For 16.0, we focused on asking as few questions as possible• Filter, view statistics or diagnoses, or explore your transaction immediately

• Locations, bandwidths, and latencies can still be specified to break down Network Transfer effects into Bandwidth, Protocol, and Congestion

• Most importantly, you can change these values whenever you need to

• Import is now broken down into use cases– Single packet trace

– Merge several traces

test

Merge several traces

– Merge traces from a WAN-accelerated environment

– Batch import

– Filter into Trace Explorer


Import a Single Capture File

test


2/10/2011

74


Merging Capture Files

• What is a merge?– Two or more traces of an application transaction can be combined

– Creates unified view of application

– Accurately determines the send and receive times of each packet

• Benefits and use cases– Allows analysis of multi-tier transactions where all traffic does not

go through one location

– Improves analysis in a production environment with congestion issues such as packet loss and retransmissions

test


Merging Capture Files (cont.)

• All traces must be of the same transaction

• All must have been captured at the “same time”

• Easier to import if there is some common traffic between tracesp

test


2/10/2011

75



File name

test

Load previously saved import details (tier names, tier locations, bandwidth, latency,…)



• Replace IP addresses with friendly names (can use DNS)

test


2/10/2011

76


After Opening/Merging

• Now what?– Filter down to a single transaction or remove user think time

– Open AppDoctor

• Note that there are only Processing and Network effects identified

– Specify bandwidth and latency to break out individual network effects

test


AppDoctor

Bandwidth, protocol and congestion are all included in one category named “Network Transfer”

To break down the various t f “N t k

testcomponents of “Network Transfer”, click the “Refine Network Effects…” button


2/10/2011

77


Refine Network Effects

Specify the local and remote bandwidths here

test Tier locations and latencies are automatically detected, but can be changed if needed


Specify Bandwidth and Latency

• Minimum bandwidth– Capacity of the slowest link

– Should be a known factor about your network

• Minimum one-way latency– PathProbe provides an estimate

– Half the value returned by ping

– Auto-calculated in a trace merge

– In many cases, AppTransaction Xpert can estimate latency of a single-sided

test

y , pp p y gtrace


2/10/2011

78


Specify Bandwidth and Latency (cont.)

• Quiz– Bandwidth of each segment is known

– Ping reports 46ms RTT

test• What would you specify for bandwidth and latency between the Client and the Server during import into AppTransaction Xpert ?

Bandwidth = 1544 Kbps Latency = 23ms


AppDoctor – After Network Definition

test

BEFORE AFTER


2/10/2011

79


LAB 5

LAB 5

Identifying Performance Problems in Unfiltered Traces

testUnfiltered Traces


Lab 5: Identifying Performance Problems in Unfiltered Traces

• Overview– Open a single side capture

– Filter traffic

– Refine network effects

– Analyze transaction

• Skills covered– Step through the import process in AppTransaction Xpert

– Understanding the importance of bandwidth settings

test

Understanding the importance of bandwidth settings



2/10/2011

80


Lab 5: Conclusion

• Imported a single-sided capture file– Imported capture file

– Filtered irrelevant traffic

– Estimated locations

– Entered local and remote bandwidth

– Estimated latency

test


Introduction to AppTransaction Xpert

2test


Day 2

2/10/2011

81


What to Expect in Advanced AppTransaction Xpert Topics

• Mix of lecture and lab

• Advanced class– Address network concepts important to using AppTransaction Xpert

– Dive into “the science” behind ACE

– Highlight new ACE features

– Discuss TCP

• Please be interactivetest

Please be interactive– Participate

– Ask questions


Prerequisites

• Prerequisites – Reviewed yesterdays material

– Reviewed session 1441

– Equivalent experience

• You should already understand the mechanics of ACE– Capturing a trace

– Importing traces

– Generating reports

test• You should already recognize “vanilla” problems– Chatty applications

– Insufficient bandwidth

– Slow servers

– …


2/10/2011

82


Agenda How AppTransaction Xpert Calculates and Predicts Components of Delay

- Lab 1: Calculating Network Delays

Understanding the different Components of Network Delay Understanding the different Components of Network Delay

- Congestion Delay

- Protocol Delay

- Lab 2: Understanding Components of Delay

Understanding Parallel Effects

- Parallel Effects

test

Parallel Effects

- User Think Time

- Lab 3: Understanding Parallel Effects


How AppTransaction Xpert calculates and predicts components of Delay

test


p y

2/10/2011

83


Components of Application Delay

• The following are key components of delay in a transaction:– Processing

• Client

• Server(s)

– Network

• Protocol (TCP WIN, Nagle’s, etc.)

• Bandwidth

• Latency (baseline, dominated by network hardware and ultimately ruled by C)

• Congestion (including queuing)

• Packet loss

test– Application itself

• Think of the top two topics as “resources”. How does the app use them?– Efficient/inefficient code

– App turns/Chattiness


Review of Network Delays

• Packets experience a complex series of – Network hops

– Technologies

– Queuing delays

– …

T1 OC3

test


2/10/2011

84


Apps See the Network as a “Black Box”, So Does AppTransaction Xpert • For most troubleshooting, complexity can be simplified

• Think of entire network as a single “equivalent” circuit

• Once packet leaves source, the delay can be separated into 3 categories

T1 OC3

– Bandwidth delay

– Latency delay

– Congestion (queuing) delay

• What about Protocol?

– Protocol delay is a function of the sender/receiver

– Packet Loss is handled by the protocol

testBandwidth, Latency,

Congestion


Bandwidth Delay

• Time it takes to stream all bits onto circuit

• This number varies with packet size

• Can be calculated from packet size and circuit bandwidthp

HeadersData

01110010010

2000 bits

2000 bits/second

test…01110010010

What is the bandwidth delay for this packet?


2/10/2011

85


Calculate Serialization Delay

How long does it take to serialize this packet on to the wire?

Serialization and BW are the same thing

test• If you know the bottleneck bandwidth across a link and the size of each

packet, you know how long it takes to get across the link due to serialization delay (ie: Bandwidth delay)

• Note: we are ignoring latency at this point


Latency Delay

• Time it takes for signal (1st bit) to travel across network– Distance / speed of light

– Normal delays caused by device forwarding

• This is a constant number for all packets

7 ms

1 ms

10 ms

1 ms

10 ms

1 ms1st bit

10 ms

testWhat is the latency for this circuit?What value would “ping” produce?


2/10/2011

86


Application Turns

• A turn is a change in the flow direction of application data

• Turns magnify circuit latency

• An application with no turns would only experience latency oncepp y p y

• An application with 99 turns would experience latency 100 times

• Simple equation for total latency delay– Latency Delay = circuit latency * (turns +1)

– This equation only works on sequential applications

No Application Turns Many Application Turnstest

pp y pp


Congestion Delay

• Queuing on devices

• Usually caused by other traffic

• Can vary from packet to packety p p

Forwarding rate = 1 packet / ms

packet

test 5 packets in forwarding queue

What is the congestion delay for this device?What is the total delay across this device?


2/10/2011

87


Determine Queuing/Congestion

Packet should have taken thi tithis time . . .

. . . but ended up taking this time

test

• If a packet should take x time but takes y time, y – x yields queuing/congestion delay


Example Network Delay

• Bandwidth = 1,000 bps

• Latency = 5 seconds

5 seconds

30 seconds

Latency delay

Bandwidth delayClient

T = 0

30,000 bits

10 20 30 40 50

test

15 seconds

Congestion delay

Server50


2/10/2011

88


Packet Trains

• Stream of packets that are sent together

• Arrive at destination spread out

• Many TCP segments of a larger application messagey g g pp g

• Spreading caused by limited bandwidth of bottleneck link

T = 0 10 20 30 40 50 60

test

10 20 30 40 50 600


Calculating Delay for Packet Trains

• Treat packet trains like 1 big packet

• Same rules apply– First—establish delay caused by bandwidth (stream bits onto wire)

– Second—add latency (last bit of the packet train getting across)

– Third—anything left over is congestion

Application Layer

testTransport Layer

Network Layer


2/10/2011

89


Calculating Delays—Advanced

• Bandwidth = 1,000 bps

• Latency = 5 seconds

+ 10

+ 5

5

Bandwidth Delay

Latency Delay

Congestion Delay

Tier Processing (Client)

+ 15

+ 5

+ 305

5

ClientT = 0 10 20 30 40 50 60 70 80 90 100

test

Tier Processing (Client)

Tier Processing (Server)

Total Delay = 45+15+20+10+10 = 100

10

10

Server

5,000 bits 10,000 bits8010 20 65 100


Pie Chart

• AppTransaction Xpert creates a pie chart that shows all of the components of delay in one screen

• Shows the total impact on response time due to each individual effect

• Each wedge is the maximum benefit that a single upgrade will provide

Bandwidth

Latency

Congestion

Client

Server

Each wedge is the maximum benefit that a single upgrade will provide

Bandwidth 45

Latency 15

Congestion 20

Client 10

Server 10

test


2/10/2011

90


Network Effects in App Doctor

• AppTransaction Xpert 16 does not require fully specifying network data for import

• Creates a “Network Effects” pie slice

test


Network Effects

• Specifying Bandwidth will cause AppTransaction Xpert to recalculate the values for the components of delay

test• For more information, see Session 1442: Application Capture and Import Strategies with AppTransaction Xpert


2/10/2011

91


Calculating Delays

• Two ways to calculate the Pie Chart– Adding up (add the components of delay)

– Squeezing down (reduce effects one at a time)

ClientT = 0 10 20 30 40 50 60 70 80 90 100

test

Server8010 20 65 100


Review of Network Delays

• Different factors can affect network delay– Bandwidth

– Latency

– Congestion

• Some factors are constant and some are variable

• Network delay is not always due to lack of bandwidth

• Important that you understand fundamentals to interpret ACE results

test


2/10/2011

92


LAB 1

LAB 1

Calculating

Network

Delaystest


Lab 1: Calculating Network Delays

• Overview– Apply knowledge of sources of delay to a real application

• Objectives– From the raw statistics calculate total bandwidth delay

– From the raw statistics calculate total latency delay

– Compare results with ACE analysis


test


2/10/2011

93


Response Time = 26.03 Network Data = 3,199,760 Bytes Size of link = 1,544,000 Kbps

Lab 1: Conclusion

p Bandwidth Delay = (3,199,760*8)/1,544,000 = 16.58 Percentage of Bandwidth = 16.58/26.03 = 63.7%

Turns = 337 Latency = 24.23ms Latency Delay = 24.23*(337+1) = 8.18 seconds P t f L t 8 18/26 03 31 4%

test Percentage of Latency = 8.18/26.03= 31.4%


Calculated that Bandwidth caused 63.7% of response time Calculated that Latency caused 29.8% of response time

Lab 1: Conclusion

test


2/10/2011

94


Why didn’t the math work out perfectly? Some amount of congestion in trace Some amount of protocol delay in trace

E ti did ’t t f bi di ti l t ffi

Lab 1: Conclusion

Equation didn’t account for bi-directional traffic

Our simple math was very close Important to perform this type of “sanity check”

test


Congestion Delay

test


2/10/2011

95


Congestion Delay

It is somewhat amazing that congestion can be calculatedVery difficult to measure and calculate manuallyACE can measure the effect even without seeing the causeACE can measure the effect even without seeing the causeACE provides 2 important pieces of information for

congestionActual congestion delay for each packet The cumulative effect of congestion on the response time

test


Key Concept

Trace file includes 1 piece of time information Time the packet arrived at the capture probe

ACE includes 2 pieces of time information for each packet Time the packet left the sending side Time the packet arrived at the receiving side

1 piece of time information per packet

test

2 pieces of time information per packet


2/10/2011

96


Key Concept

ACE knows both the send and receive time Delay analysis requires that both values are known How does ACE derive the second piece of time information?p Trace Merge Single Trace Adjust

test


Trace Merge

Each trace shows what happened locallyOnce clocks are aligned, draw the packets from send to receive

Client

Trace captured at client side

BA C C* D

Trace captured at server

ClientT = 0 2 3 4 5 61 7 8 9 10

A C C*B DT = 0 2 3 4 5 61 7 8 109

test

Server

Trace captured at server side

BA C* D Server0 2 3 4 5 61 7 8 9 10

B DA C*

0 2 3 4 5 61 7 8 109


2/10/2011

97


Single Side Adjust

With a single trace, ACE only knows what happened locallyMust estimate what happened on network and at other side Based on bandwidth and latency values set during import

ACE ti t h th k t t ACE estimates when the packet was sent ACE estimates when the packet will arrive

Client


ClientT = 0 2 3 4 5 61 7 8 9 10

A B C C* E

T = 0 2 3 4 5 61 7 8 9 10

test

Client

BA C C* E

Server0 2 3 4 5 61 7 8 9 10


Trace Merge

What happens if the import parameters are incorrect Bandwidth is set too low Latency is set too highy g

Might create false packet crossingsThis will confuse analysis

Client


ClientT = 0 2 3 4 5 61 7 8 9 10

A B C D E

T = 0 2 3 4 5 61 7 8 9 10

testBA C D E

Server0 2 3 4 5 61 7 8 9 10


2/10/2011

98


Key Concept

With BOTH send and receive time ACE knows Actual packet delayOrder of how packets were sent and receivedp

What would happen if we graphed the delays?

testTotal Packet Delay


Packet Size vs. Delay Graph

• Graph the packet size vs. total packet delay

Del

ay (s

ec)

Congestion

Slope is bandwidth*

test

Packet Size (bits)

Pure latency

* Inverse of bandwidth (1/BW)


2/10/2011

99


Packet Size vs. Delay Graph (cont.)

• Graph Packet Size vs. Total Packet Delay

Del

ay (s

ec)

test

Packet Size (bits)


Viewing Congestion in ACE

• Several good ways to view congestion in ACE– AppDoctor diagnosis

– Summary of Delays

– Congestion drill-down

– Data Exchange Chart

test


2/10/2011

100


Congestion in the AppDoctor Diagnosis

• Specifically marks category as a Bottleneck

test


Congestion in the Summary of Delays

• Shows the cumulative effect of the congestion on response time

• How much slower was the application because of congestion?

test


2/10/2011

101


Drilling into Congestion

test• Drilling into congestion can show if the packets are getting queued or

otherwise slowed down


Drilling into Congestion (cont.)

test• Look for “plateaus”

• Always keep in mind the scale of the chart!

• Is delay uniform or random?– Is there any encryption on the link?


2/10/2011

102


Congestion in the Drill-Down

• Packet Congestion graph

• Retransmission graph

test


Congestion in the DEC

• Network chart layered with Packet Congestion graph

test


2/10/2011

103


Congestion Review

• Congestion Delay is cause by other traffic on the network

• Always better to merge traces from congested circuits– Congestion values for a trace merge are precise

– Congestion values for a single-side adjust are approximations

• Several ways to see congestion and its effect on the application

• Imagine “squeezing” the congestion out of the trace to calculate its contribution to overall delay

test


Protocol Delay

test


2/10/2011

104


Protocol Delay

• Caused by the artificial throttling of traffic

• Packets are prevented from even getting onto the network

• Caused mainly by TCPy y

test


Why Does TCP Exist?

• TCP offers reliable data transport to applications– Application data is guaranteed to be delivered

– Packets that are “lost” by the network will be retransmitted

• TCP makes applications behave in a network-friendly manner– Throttles the rate at which application sends data

– Ensures that no single user consumes all of the bandwidth

• Both can impact application performance

test


2/10/2011

105


What Does Protocol Delay Look Like?• Protocol delay prevents packets from even getting on the network

Application Layer

Transport Layer

Network Layer

Protocol DelayNo Protocol Delay

test

0 2 3 4 5 61 7 0 2 3 4 5 61 7 8 9 10 11 12 13

Protocol Delay


Observe Protocol Behavior

Protocol delay is when the ysending side delays or throttles sending of data (this is different than processing delay)

test• Observe how data leaves in stages indicating Protocol Delay

• Observe other attributes like TCP window size

• Our example here is ridiculously simplified, but real protocol delay is observable in a real trace


2/10/2011

106


Forms of Protocol Delay

• Critical that you understand mechanisms of TCP

• Several different TCP mechanisms cause can protocol delay– TCP windowing

– Slow Start

– Frozen window

– Nagle’s algorithm

– Retransmissions

– Out of sequence packets

test


Viewing Protocol Delay in ACE

• Several ways to view Protocol Effects –AppDoctor diagnosis

S f D l–Summary of Delays

–Protocol drill-down

–TCP Visualization in DEC

test


2/10/2011

107


Protocol Delay in the AppDoctor Diagnosis

• Specifically marks category as a Bottleneck

test


Protocol Delay in the Summary of Delays

• Double-click on slice to get more detail

test


2/10/2011

108


Protocol Delay Drill-Down

• Graphs dynamically appear based on AppDoctor Diagnosis

• Users can manually add or remove graphs

• Protocol Delayy– TCP In-Flight Data

– Retransmissions

– Out of Sequence Packets

– TCP Nagle’s Delay

– TCP Advertised Receive Window

test


Protocol Delay Review

• Protocol delay results from the artificial throttling of traffic

• Several different TCP mechanisms can cause it

• Prevents the packet from even getting onto the network p g g

• Can often be calculated without a trace merge

• Imagine “squeezing” the protocol effects out of the trace to calculate its contribution to overall delay

test


2/10/2011

109


LAB 2

LAB 2

Understanding

Components of

Delaytest


Lab 2: Understanding Components of Delay

• Overview– User is transferring files with Microsoft file browser

• Transfer time is very slow

• Want to understand cause of delay

• Objectives– Identify and understand the sources of delay

– Compare slow transfer to fast transfer


test• Homework– Continue with homework sections of lab manual if you have time


2/10/2011

110


Lab 2: Conclusion

• File transfer is slow• Primary source of delay is protocol/congestion• Some “potential” problems with TCP but no smoking gun• Look at packet delay graph to evaluate congestion

• Significant congestion on WAN circuit– Base latency is 25ms– Some packets take 125 ms to cross the network

test


Lab 2: Conclusion

• Congestion is primary cause of delay

• Removing other traffic on network fixes problem– Reduces congestion

– Response time decreases

• Might be a case for QoS

Significant Congestion No Congestion

test


2/10/2011

111


Parallel Effects

test


The “Mysterious” Gray Wedge

• AppTransaction Xpert shows “parallel effects” as a gray pie wedge

test– Transaction has more than one effect taking place at one time

– HTTP is an example of a transaction that exhibits parallelism


2/10/2011

112


Parallel Effects Drill-Down

• Can show the individual effects with their combined parallel effects together

• Solid bars are the individual effects

• Striped bars are the parallel effectsp p

• Optimizing both components simultaneously will give the benefits of the individual effects AND the parallel effects

T = 0 10 20 30 40 50 60 70 80 90 100

Response Time

Parallel Effects of Bandwidth and Latency

Bandwidth 60%

Latency 10%

Server 20%

Parallel 10%

test


Parallel Effects of Server and Latency



• Parallel delays may seem counterintuitive at first

• It is a real effect and does happen in the real world

• Real Life Parallel Effects example that you will remember!p y– Reading a paper while eating breakfast

– Cooking a meal (dessert in the oven while steak is on the grill)

• Would be difficult to split these into simple 2 category pie charts– Incorrect to say grilling steak was 50% and dessert was 50% ?

– Suggests that eliminating dessert would cut dinner preparation in half

– Probably more like 25% steak, 25% dessert, 50% both

– Life is parallel

test

Life is parallel


2/10/2011

113


Types of Applications

• Applications fall into two main categories–Transactional

•St d ff t l ti hi b t k t•Strong cause and effect relationship between packets

•Each application message is dependent on the one before it

•Examples– Database queries

– Web transactions

– File transfers

–Asynchronous

test•No strong cause and effect relation between packets

•Examples– Voice

– Citrix



• Transactional application can be further divided–Sequential

•Si l ti th•Single execution path

•Application is only doing one thing at a time

–Parallel•Two or more simultaneous execution paths

•Application is doing multiple things at a time

test


2/10/2011

114



• Sequential (Adding up method)– Each slice of time can be placed into a single delay category

– The components of delay add up to the total response time

ClientT = 0 10 20 30 40 50 60 70 80 90 100

test

Server



• Parallel (Squeezing down method)– Some slices of time could be placed in multiple delay categories

– The components of delay would up to more than the total response time

ClientT = 0 10 20 30 40 50 60 70 80 90 100

testServerServer

Connection 1Connection 2Connection 3


2/10/2011

115


AppDoctor Individual Effects

Response Time 6 sec

Server 2 sec (33%)

Client 2 sec (33%)

• Summary of Delays for a sequential transaction

( )

Bandwidth 2 sec (33%)

test


AppDoctor Parallel Effects

• Summary of Delays for a parallel transaction

• Server is now processing while simultaneously sending the large message

test


2/10/2011

116


AppDoctor Parallel Effects (cont.)

How does AppDoctor handle multiple effects occurring at the same time?

Response Time

6 sec

Server 2 sec (33%)

test

(33%)

Client 2 sec (33%)

Parallel Effects

2 sec (33%)

The "Parallel Effects" categoryMixed wedge shows total impact due to server processing and bandwidth occurring in parallelBecause there may be many mixed wedges, ACE shows these as gray in the Pie Chart


Understanding Parallel Applications

• Calculating delays for sequential transactions is fairly easy– Each slice of time can be attributed to a single cause

– All the slices add up to the total

• Calculating delays for parallel transactions can be complex– A slice of time may be attributed to multiple sources

– Only the squeezing method works here

• Parallel Effects are subtle and not always easy to visualize

• Parallel Effects do not always occur just because delays overlap

• Aren’t you glad you have AppTransaction Xpert to do this for you

test


2/10/2011

117


Parallel Effects Example

• Parallel– Two independent connections occur in parallel

– Both must complete for the application to complete

ClientConn. B

A

B B B

testServerServer

Conn. A



• Slide Conventions– Show independent connections as parallel DECs

– Keep in mind that these are happening on top of each other

– Obvious now why you can’t just add delays up

Client

ServerServer

T = 0 10 20 30 40 50 60 70 80 90 100

Connection A

test Client

ServerServer

T = 0 10 20 30 40 50 60 70 80 90 100

Connection B


2/10/2011

118



• Parallel applications with squeezing method– Perform experiments that squeeze out each category of delay

• Color associated with that delay would disappear

• Response time of transaction may decrease

– Record the difference from each experiment

Client

ServerServer

T = 0 10 20 30 40 50 60 70 80 90 100

test Client

ServerServer

T = 0 10 20 30 40 50 60 70 80 90 100



• What would happen if bandwidth became infinite?– Green would be squeezed out

– Total response time of the transaction would change to 40 seconds

– Bandwidth is responsible for 60 seconds of response time (60%)

Client

ServerServer

T = 0 10 20 30 40 50 60 70 80 90 100

test Client

ServerServer

T = 0 10 20 30 40 50 60 70 80 90 100


2/10/2011

119



• What would happen if latency became zero?– Red would be squeezed out


– Latency is responsible for 10 seconds of response time (10%)

Client

ServerServer

T = 0 10 20 30 40 50 60 70 80 90 100

test Client

ServerServer

T = 0 10 20 30 40 50 60 70 80 90 100



• What would happen if the server became infinitely fast?– Yellow would be squeezed out


– The server is responsible for 20 seconds of response time (20%)

Client

ServerServer

T = 0 10 20 30 40 50 60 70 80 90 100

test Client

ServerServer

T = 0 10 20 30 40 50 60 70 80 90 100


2/10/2011

120



• Something interesting just happened– Bandwidth caused 60%, Latency caused 10%, Server caused 20%

– The individual sources of delay are only responsible for 90%

– What caused the remaining 10%?

• Delays that are not removed by a single source are Parallel Effects

Bandw idth

Bandwidth 60%

Latency 10%

testLatency

Server

Parallel

Latency 10%

Server 20%

Parallel 10%



• Squeeze out 2 effects at the same time– Bandwidth = 60%

– Latency = 10%

– Server = 20%

– Parallel Effects = 10%

Client

ServerServer

T = 0 10 20 30 40 50 60 70 80 90 100

Thread A

test Client

ServerServer

T = 0 10 20 30 40 50 60 70 80 90 100

Thread B


2/10/2011

121



• Remove the Server and Latency together– Server by itself is 20%, Latency by itself is 10%

– The pair is responsible for 35 seconds of response time (35%)

– Parallel interaction of the 2 provides a bonus benefit if changed simultaneously

Client

ServerServer

T = 0 10 20 30 40 50 60 70 80 90 100

Thread A

test Client

ServerServer

T = 0 10 20 30 40 50 60 70 80 90 100

Thread B



• Remove Bandwidth and Latency together– Bandwidth by itself is 60%, Latency by itself is 10%

– The pair is responsible for 75 seconds of response time (75%)

– Parallel interaction of the 2 provides a bonus benefit if changed simultaneously

Client

ServerServer

T = 0 10 20 30 40 50 60 70 80 90 100

Thread A

test Client

ServerServer

T = 0 10 20 30 40 50 60 70 80 90 100

Thread B


2/10/2011

122


Parallel Effects Drill-Down

• Can show the individual effects with their combined parallel effects together

• Solid bars are the individual effects

• Striped bars are the parallel effects

• Optimizing both components simultaneously will give the benefits of the individual effects

Bandwidth 60%

Latency 10%

Server 20%

Optimizing both components simultaneously will give the benefits of the individual effects AND the parallel effects

Response Time

T = 0 10 20 30 40 50 60 70 80 90 100


testParallel 10%

Parallel Effects of Server and Latency


Parallel Effects Summary

• Delays that can not be removed by a simple change of one parameter

• To reduce, you must change at least two parameters simultaneously

• These delays are not bad y– Application is efficient

– Application is doing multiple things at the same time

testBandw idth

Latency

Server

Parallel


2/10/2011

123


User Think Time

test


Capturing a Good Transaction (Task)

• Important to get an “atomic” transaction– Atomic transaction is the smallest transaction that you can capture effectively

– Capture should be from edge of user input to screen refresh

– Eliminate “PEBCAK”

test


2/10/2011

124


Atomic Transaction Example

• Modify Customer Record– Login

• Call up the Login screen• End user complains• Type information

• “Submit”

– Look up record• Call up correct screen/menu

• Type information

• “Submit”

– Modify data• Mouse/cursor around

End user complains “modifying a customer record is slow”

• Need to find out what is involved in transaction

test• Type information

• “Submit”

– Logout• Call up Logout screen

• “Submit”

Red arrows indicate the key part of the transaction to capture


Identifying and Removing User Think Time

• Always get as close as you can to a perfect “Atomic” transaction as described in the Capture Section p

• However, sometimes you just can’t get the perfect capture

• “User think time” enters the trace

• AppTransaction Xpert deals with think time– Auto detects when calling up AppDoctor

– Allows user to explicitly specify think time

test


2/10/2011

125


Removing User Think Time

AppTransaction Xpert automatically identifies that the trace contains user think time. This dialog comes up when AppDoctor is selected

AppTransaction Xpert

testalso allows the user to pick any dependency on the client (first talker) and categorize it as user think time


Removing User Think Time

Original Analysis

testUser Think Time Removed


2/10/2011

126


LAB 3

LAB 3

Understanding

Parallel Effects

test


Lab 3: Understanding Parallel Effects

• Objectives– Explore an application transaction that has significant parallel effects

• Trace is fairly simple

• Trace is somewhat artificial

– Practice visualizing the squeezing method to understand the behavior

– Use Drill-down and QuickPredict to understand the interactions


test


2/10/2011

127


Lab 3: Conclusion

• Parallel delays may seem counterintuitive at first

• It is a real effect and does happen in the real world

• This example is exaggerated to make it easy to visualizep gg y

test


Agenda Overview of TCP

- TCP Basics

- TCP Windowing

- Lab 4: Examining TCP Behavior

T bl h ti TCP Troubleshooting TCP

- Multi-User QuickPredict

- Bandwidth-Delay Product

- Lab 5: Troubleshooting a Slow File Transfer

- Detecting Packet Loss

- Frozen Window

- Nagle’s Algorithmtest

- Nagle s Algorithm

Discrete Event Simulation

Instructor Demo: Nagle’s Algorithm using DES

References

- What’s New ?

- Wrap up255 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc.

2/10/2011

128


TCP Basics

test


What is TCP?

• Transmission Control Protocol

• Transport layer protocol p y p

• Layered between IP and application layer

• Typical applications such as

– HTTP (Web)

– POP, IMAP, SMTP (e-mail)

Application Application

TCP

IP

test– Telnet, FTP, and more

Data Link


2/10/2011

129


Why Does TCP Exist?

1) TCP offers reliable data transport to applications

• User data is guaranteed to be delivered

2) TCP makes applications behave in a network-friendly manner

• TCP may throttle the rate at which it sends user data

Both impact application performance

test


Which Applications Don’t Use TCP?

• TCP adds overhead

• TCP limits throughput

• Not all applications need what TCP providespp p– VoIP – Voice over IP

– Video Conferencing

– SNMP – Simple Network Management Protocol

– Network Gaming Protocols

– Domain Name System (DNS)

– Network Time Protocol (NTP)

test


2/10/2011

130


Motivation – Why Is TCP Important?

Example TCP/IP Stack

Application Developers HTTPSOAP

HTTPSOAP

IPFIBER

IPETHERNET

IPETHERNETNetwork Managers

TCP TCP

Application Developers

SSLHTTP

SSLHTTP

test• TCP is in a key location of networked applications’ protocol stacks

– Frequently misunderstood– This makes people and software that understand TCP valuable


Review: ACE Application Message Chart

Timeline

Application message groups

Application message arrows

Tiers

test

Red packets have no payload (usually acknowledgements, SYNs, or FINs)

Packet sizes

g p


2/10/2011

131


How to Read TCP Decodes

Size is size of entire packet (including IP, MAC, etc.)

Receive window

Special flagsSequence number

Destination port

Receive window

F b t k ftest

Acknowledgement number

Payload length

Source portFrame number taken from trace file(s). May not always be in strict order due to trace merge


Opening a Connection

• Three-way handshake to open connection

• Used to exchange– Initial SEQ numbers

Cli– TCP Options Client

1A

2

3C

test

Server

2B


2/10/2011

132


Closing a Connection

• Four-way handshake to close connection• “I have no more data to send.”• Typical example

client273B

274C

connection closed at client

test

server

272A

B C

275D

connection closed at server


Closing a Connection (cont.)

• Often only three segments used

• Second FIN also is an acknowledgement for first FIN (B&C in one packet)

• TCP Resets are analternate way toterminate a connection 86

A

87B&C

88D

test

B&C


2/10/2011

133


SEQ and ACK Numbers

• Ensures that data was received

• Sending TCP gives each segment a sequence number– SEQ of segment K is SEQ of segment K-1 plus length of segment K-1

• Receiving TCP acknowledges reception by replying with ACK– ACK number is the SEQ number of the next expected byte

• SEQ and ACK numbers are inbytes, not packets

Receiver

SEQ=100 LEN=10

Sender

test

Q

ACK= ?

SEQ = ? LEN=55

110

110

K-1 K

Seq # K-1 Seq # K


SEQ and ACK Quiz

• Which frame(s) is 36 ACKing?

• Which frame(s) is 44 ACKing?

test


2/10/2011

134


TCP Acknowledgements

• Receiver normally ACKs every other packet– Inefficient to ACK every packet

• What if only one packet is received?– Delayed-ACK timer must expire

– 100-200 ms (OS determines exact delay)

– Usually does not impact total application performance

emen

t

test

Dat

a

Ack

now

ledg

e268 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc.

Review: TCP Basics

• TCP is a connection-based protocol

• It uses a three-way handshake for opening connections

• Each side closes the connection separatelyp y

• SEQ and ACK numbers ensure reliable delivery

test


2/10/2011

135


TCP Windowing

test


TCP Windowing

• TCP window size is the maximum amount of data TCP will allow outstanding (“in-flight”)

• If window size is less than or equal to outstanding data, TCP will not send data

• In trace below, server is waiting for ACKs before sending more data

Client

test

Server


2/10/2011

136


TCP Windowing

• TCP uses the minimum size of the three windows– Congestion control window

• Changes during connection

• Internal to sending TCP

• Not visible in real world

• Lives in the “mind” of the sender

– Receive buffer

• May change during connection

• User-configurable

• Visible as WIN= in decodes

test– Send buffer

• Controlled by sending

application

• Not visible in decodes

• Rarely the bottleneck

Congestion Control Window

Receive Buffer

In-Flight Data


TCP Congestion Control Window and Slow Start

• Estimate of how much data the network can handle

• Sending TCP increases the congestion control window each time it receives g gan ACK– Congestion control window increases exponentially every round-trip time

• Actually stored in bytes: 1460, 2920, 5840, 11680

• When TCP first starts transmitting data, it goes slowly– Sender starts by sending one segment, then waits for ACK

test


2/10/2011

137


TCP Congestion Control Window in Slow Start

CWND = 1Data (S1)

Sender Receiver

CWND = 2

CWND = 3

CWND 4

ACK (S2)

Data (S3)Data (S2)

Data (S5)Data (S4)

Data (S6)

ACK (S4)

testCWND = 4

CWND = 7

ACK (S8)

Data (S7)Data (S8)Data (S9)

ACK (S10)Data (S10)

ACK (S12)

Data (S11)Data (S10)


Bulk Data Transfer

• Congestion window is greater than receive window

• TCP sends data as soon as it receives acknowledgements

• In-flight data is constantg

• In-flight data = outstanding data = unacknowledged bytes

• Data transfer limited by receive window

test

When you see a pattern like this, blame TCP275 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc.

2/10/2011

138


Review: TCP Windowing

• TCP limits the amount of outstanding data

• Minimum of three windows is used– Send window: How much data the sender can handle

– Receive window: How much data the receiver can handle

– Congestion control window: How much data the network can handle

• Congestion window– Initially small (1, 2, or 4 segments)

– Grows exponentially during slow start

test


LAB 4

LAB 4

Examining

TCP Behavior

test


2/10/2011

139


Lab 4: Examining TCP Behavior

• Use AppTransaction Xpert to examine TCP behavior

• You will see– Connection opens

– Slow start

– Congestion control window

• Raise your hand, and a TA will assist you with any questions


test


Lab 4: Conclusion

• TCP uses SEQ and ACK numbers to ensure reliable delivery

• Minimum of three windows (congestion, receive, and send) determines whether TCP sends more data

• Comparison of bulk and chatty applications– Window size and bandwidth determine throughput for bulk applications

– Latency determines throughput for chatty applications

test


2/10/2011

140


When to Suspect TCP as a Bottleneck

• You should suspect TCP of causing problems when:– You cannot explain why an application is running slowly

– The servers and network are adequately provisioned

• Servers have low load

• Sufficient bandwidth

• Links not congested

– The ACE Live™ Response Time Composition Chart shows

• Large Payload Transfer (Xfer) Times

• Large Retransmission Delays

– AppTransaction Xpert ’s AppDoctor Summary of Delays tab shows

test• Large Network Transfer Effects

• Large Protocol delays

– AppTransaction Xpert ’s AppDoctor Diagnoses point to

• One or more TCP-related bottlenecks or potential bottlenecks


Troubleshooting TCP with ACE Live- Excellent for focusing troubleshooting, monitoring TCP health

• Where to start looking, where and when to capture• Measures connection resets, packet loss, long response times, and more

ACE Live Response Time Composition Chart

– Large Retransmission DelayLarge Retransmission Delay

• May be sign of packet loss

• May also be unfiltered duplicate packets

– Large Payload Transfer Times

• May be TCP-related

• May also indicate bandwidth, congestion or even server processing causes

test


2/10/2011

141


Troubleshooting TCP with AppTransaction Xpert

• AppDoctor provides quick way to check for TCP problems

• Suspect TCP if the Summary of Delays shows– Large Network Transfer Effects

– Large Protocol delays

• Use diagnosis tab to identify whichspecific TCP-related issue affectsyour trace– Bottlenecks

– Potential bottlenecks

test


Understanding Network Transfer Effects

• Combination of three effects related to transferring data– Bandwidth, Congestion, Protocol

• Bottleneck bandwidth is needed to separate these phenomena– Refine Network Effects key to identification of delays due to the TCP protocol

test


2/10/2011

142


How to Identify TCP Problems

• Look at the AppDoctor Diagnosis tab– Effect of Protocol

– Effect of Network Transfer

– Last 7 stats deal with TCP

– Helps target the specificTCP issue

• Look at decodes for long

application transactions

• Look in the networktest

• Look in the network

chart at the start and

end of long delays


How to Identify TCP Problems (cont.)

• Double-click in the Summary of Delays for more information

• Use separate drilldown screens for Protocol and Congestion

test


2/10/2011

143


How to Identify TCP Problems (cont.)

• Congested networks also can cause protocol delays• TCP implements congestion avoidance• Single-sided captures of receiving data may make congestion look like

protocol delay or tier processing

test


Review: Analyzing TCP Behavior

• Look for TCP problems in traces with large amount of protocol delay

• Congestion delays also induce protocol delays

• AppDoctor (forensic)pp ( )– Separately diagnoses protocol and congestion

– Diagnoses several TCP problems

• QuickPredict (what if?)– Separately models protocol and congestion

– Can simulate changing TCP Window Size

test


2/10/2011

144


Review: ACE QuickPredict• Quick simulation engine

– Combination of analytical and discrete event simulation– Uses analytical models of TCP behavior

• Initially shows a pristine view of network• Allows you to study effects of the network on the applicationAllows you to study effects of the network on the application• Outputs graphs of response time vs. one of the following:

– Bandwidth– Latency– Packet loss– Link utilization– TCP receive window size– NEW: WAN Acceleration parameters

test• QuickPredict Bar Charts supports

asymmetric window sizes


Review: QuickPredict Input and Output

• Y axis is always response time

• X axis is a range of values for one of:– Bandwidth, Latency, Packet Loss, Link Utilization, TCP Window size,

and WAN Acceleration (NEW!)

Change values of other variablesto see impact on response time

test


2/10/2011

145


Multi-User QuickPredict

test


Multi-User QuickPredict (MUQP)

• Single-user QuickPredict considers one client only

• MUQP allows you to have more than one– Client and client site

– Applications (either ACE traces or ACE Whiteboard traces)

– Data centers

• Output– Response time (including

– delay components)

– Utilization

• Results

test

Results– Average

– N-th percentile


2/10/2011

146


Inputs to MUQP

• Deployed applications (defined in ACE and/or ACE Whiteboard files)

• Network topology (number of client sites, clouds, and data centers)

• Tier-pair parameters (bandwidth, latency, link utilization, window size, and p p ( ypacket loss)

• Application deployment options (number of transactions per hour for each client site)

test


Outputs of MUQP

• "Average-case" response times for applications

• "Worst-case" results for access links (utilization) and applications (total response time and components of tier processing, latency, protocol, and congestion delay)

test


2/10/2011

147


Instructor Demo of MUQP

test


Bandwidth-Delay Product

test


2/10/2011

148


Bandwidth-Delay Product• Bandwidth-delay product — Defined as amount of data (in bytes) that can be

sent before ACK is received for the first unacknowledged bit of data – 2 x one-way latency x bandwidth of slowest link

• To “fill the pipe,” TCP window size must be equal to bandwidth-delay product• If window is too small, TCP must wait for acknowledgment before sending new datag g• Example below

– Bandwidth: 1,544 kbps = 193 KB/sec– Latency: 250 ms = 0.250 sec– BWDP: 2 x .25 x 193 = 96.5 KB– Receive Window: 8 KB (1/12 ideal size)– Throughput limited to: 130 kbps (1/12 link speed)

test


Bandwidth Delay Product = 2 × Capacity = 24 KB (16 segments)

Capacity = .096 s × 125 KB/s = 12 KB (8 segments)

Bandwidth-Delay Product

Sender Receiver

Bandwidth-Delay Product = 2 × Capacity = 24 KB (16 segments)

96 ms

test

1 Mbps


2/10/2011

149


LAB 5

LAB 5

Troubleshooting

a Slow File Transfer

test


Lab 5: Troubleshooting a Slow File Transfer

• Overview – Computer manufacturer moving bulk data from Chicago to Taipei has dedicated

T1 without any packet drops

– Not seeing expected response time

– You are a network engineer

– Your manager: “It must be the network”

– Contemplating (expensive) link upgrade

• Objective– Determine the cause of slow

response time

test– Propose and test a solution



2/10/2011

150


Lab 5: Conclusion

• Simple TCP configuration change improved performance

• Over same time period:p– Large receive window

test– Small receive window

• What is the downside of making this change?


Changing the Receive Window

• How do I change the receive window on my computer?– Windows

• http://www.microsoft.com/technet/network/tcpip/default.mspx

• http://www.microsoft.com/technet/itsolutions/network/deploy/depovg/tcpip2k.mspx

• Google search: windows tcp/ip implementation details

– Solaris

• http://www.sean.de/Solaris/soltune.html#water

– Linux

• http://www.speedguide.net/read_articles.php?id=121

test• Default for modern operating systems is 64 KB

– Maximum advertised receive window value in TCP packet is 64 KB

– How do we handle today’s large capacity, high latency pipes?

• TCP Window Scaling


2/10/2011

151


TCP Window Scaling

• TCP allows for WIN= field to be scaled by a power of 2– Decodes will show WIN<<2=262140– WIN<<x means multiply field by 2x

• This setting is exchanged in TCP options at beginning of connectionThis setting is exchanged in TCP options at beginning of connection– Both sides must agree and understand this option

test

• What if a packet trace does not contain the opening of a connection?– AppTransaction Xpert cannot detect the presence of TCP Window scaling


Detecting Packet Loss

test


2/10/2011

152


TCP and Packet Drops

• Where do packets get dropped, and why?

– Media collisions

– Wireless signal interference

– Congested networks

• Endpoints must react to overloaded routers

– Drops are a form of indirect communication between routers and senders

• Routers have multiple strategies for keeping queue size in check

– Tail drop

– Random Early Detection (RED)

test


Packet Loss and Retransmissions

Packet is dropped by the network and retransmitted Time between when it should have arrived and when it did arrive Controlled by TCP timers

C t d P t l D l Counted as Protocol Delay Retransmission also invokes congestion control Slows down throughput Counted as Protocol Delay

ClientT = 0 2 3 4 5 61 7 8 9 10

A B B

test

Server0 2 3 4 5 61 7 8 9 10

Protocol Delay


2/10/2011

153


Drops and Retransmissions

• TCP must make sure all bytes of a stream are received– TCP guarantees delivery by retransmitting packets presumed dropped

• Two methods by which TCP will retransmit a dropped packetTwo methods by which TCP will retransmit a dropped packet

–Retransmission timeout• Sending TCP schedules a timer for outstanding packets

• Timer is cancelled when the packet is acknowledged

• If timer expires, packet is presumed missing, re-sent

–Triple duplicate ACKs• ACK number indicates next expected packet

test• If receiving TCP does not get the expected packet, it ACKs every packet

• If receiving TCP does not get the expected packet, ACK number does not increase.

• If sending TCP receives acknowledgements with the same ACK number, it knows a packet was likely lost


TCP One Loss: Retransmission TimeoutSender Receiver

Data (S104)


ACK (S103)Data (S104)

ACK (S105)

Data (S108)

Data (S107)

Data (S106)

Data (S105)

( )

ACK (S106)

test

Data (S106)

ACK (S109)

Retransmission timeout

ACK (S106)


2/10/2011

154


TCP One Loss: Triple Duplicate ACK

Sender Receiver


ACK (S103)

Triple duplicate ACK and fast

Because 104 was not received, Receiver does not change ACK number

Data (S104)

ACK (S104)

Data (S107)

Data (S106)

Data (S105)

ACK (S103)

ACK (S104)

ACK (S104)

Data (S108)

test

ACK and fast retransmit

Data (S104)

ACK (S109)

( )

ACK (S104)


Cost of Loss Detection

• Retransmission timeout– There are not enough ACKs for triple duplicate ACK– Cost: Timer (often large) plus time to send 1 packet– Congestion window decreases to one segment

• Triple-duplicate ACKs– Cause fast retransmit– Cost: Time to send 1 packet

Triple-duplicate ACK

Retransmission

test

• Notice the send times Retransmission


2/10/2011

155


Review: Detecting Packet Loss

• Two mechanisms– Retransmission time-out (RTO)

• Costly

• Always works

– Triple-duplicate ACK

• Relatively efficient

• Does not always happen

• Chatty applications rarely “qualify”

• Food for thought– Why acknowledge every packet during loss?

test– Why not quintuple-duplicate ACK?

– Why not single-duplicate ACK?


Frozen Window

test


2/10/2011

156


TCP Frozen Window

• Receiver stores packets in receive buffer

• Advertised receive window “WIN=” reduced by data in buffer

• Normal applications remove data quicklypp q y

• What happens if:– A server is handling too many clients?

– What if the server is running other applications?

– What if a another thread is hogging the CPU?

test


TCP Frozen Window (cont.)

• As WIN decreases, buffer fills up, sender cannot send data

• Connection becomes “frozen”

• Sender will send probe packets to see if window size has increasedp p

test


2/10/2011

157


TCP (Almost) Frozen Windows

• As WIN decreases, sender cannot send data

• What if the WIN is less than MSS?

• Sender could send data, but often does not

• Detection of “chilly windows” new to AppTransaction Xpert 16.0

test


Drilling into Protocol (cont.)

test

• Frozen Windows:– Receiving WIN goes to zero– Sending side sends “probe” packets


2/10/2011

158


Nagle’s Algorithm

test


Nagle’s Algorithm

• Nagle’s Algorithm– TCP mechanism to reduce sending of many tiny messages

– Sender may not send data unless a full, MSS-sized segment can be sentOR there is no unacknowledged data

• Delayed Acknowledgement

• Interaction of these two mechanisms sometimes causes problems

• Ridiculous situation where both sides wait for the other to do somethingClient

testB

Server

A B

200 ms317 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc.

2/10/2011

159


Nagle’s Algorithm Example

• Client sends segment A (less than MSS)

• Client wants to send segment B, but needs to wait for ACK

• Server is waiting for another segment before sending ACKg g g

Eventually, server sends ACKClient then sends segment BProcess repeats…

Client

testB

Server

A B

200 ms318 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc.

Nagle’s Algorithm Example (cont.)

• What causes the delay for this application?

test


2/10/2011

160


Nagle’s Algorithm and Delayed ACK

• To eliminate delays when sending consecutive small messages:– Rewrite the application

• Sending multiple small consecutive messages is inefficient

• Sometimes you can’t change the application

– Disable Nagle’s algorithm

• Done within the application

• Effects only the application changed: (TCP_NODELAY socket option)

• Not always possible to change an existing application

– Acknowledge every packet

• Practical real world alternative

test• On Windows: http://smallvoid.com/article/winnt-nagle-algorithm.html

• Warning: Applies to every application on the machine!– Interacts with congestion control window, affecting bulk applications


References

• TCP in OPNET– Online documentation– Overview PDF

• Model Library/Standard / TCP Model Usage GuideACE TCP Wi d i B ttl k li d t ti– ACE TCP Windowing Bottleneck online documentation

• Applications in OPNET– Several whitepapers including SMARTE

• Help / Product Documentation / Modeling Methodologies / Methodologies and Case Studies

• http://www.psc.edu/networking/projects/tcptune/– Setting system wide TCP window sizes– Listing of TCP features for each operating system

• http://www.ncsa.uiuc.edu/People/vwelch/net_perf/tcp_windows.htmlInstructions on setting TCP window sizes within application code

test

– Instructions on setting TCP window sizes within application code• http://www.sean.de/Solaris/soltune.html

– Tuning your TCP/IP stack on Solaris• TCP/IP Illustrated, Volume I, Richard Stevens


2/10/2011

161


Take-Away Points

• Windowing causes TCP to wait before transmitting application data

• Bulk data-transfer throughput often limited by TCP windowing

• TCP frequently does not use the full available bandwidthq y

• Packet loss can cause significant reduction in throughput

• QuickPredict is a fast, easy, accurate way to predict application response time

• Use DES to model and study detailed behavior

• Simulate using the proper TCP machine settings

Next step: Run homework labs for a greater understanding of TCP

test


What’s New ??

test


2/10/2011

162


What’s New in AppTransaction Xpert ??

• 64-Bit Support– Access more that 4GB of memory on 64-bit machines which allows you analyze

much larger capture files

• Windows 7 Professional (32-bit and 64-bit)

• Capture Enhancements– Manage captures from ACE Live appliances within the Application Capture

Manager. New ACE Live tab to quickly download and preview data

– ACE Capture Agent 3.9 Ability to filter the traffic data by time range, IP addresses, and/or IP conversations.

– Capture File Filters Greater flexibility in specifying hosts and ports.

test• Support for Riverbed Transparent Mode

– The AppTransaction Xpert import now supports capture files from Riverbed accelerators running in full or port transparency mode


What’s New in AppTransaction Xpert (cont.)

• Streamlined Import Process– Quickly open capture files through the user interface without specifying any information

(such as location, bandwidth, latency, etc.)

• Bandwidth Latency and Locations can be specified as neededBandwidth, Latency, and Locations can be specified as needed

– Automatic Filtering of Duplicate Packets and TSO (TCP Segmentation Offloading)

– Simplified Manual Merge Workflow

• QuickPredict Enhancements - WAN Acceleration Support– Support for WAN optimization technologies by allowing users to predict behavior for

compression, caching, and application acceleration

• HTTP Anal sis Enhancementstest

• HTTP Analysis Enhancements– Automatic detection of HTTP on non-standard ports (thus eliminating the need to perform

redecodes or modifying config files)

– Improved analysis for packet slicing and missing packets

– HTTP-specific visualization of object response times in TreeView


2/10/2011

163


References

test


Additional Documentation• Online at:

www.OPNET.com / Support / Support Center

–Documentation

–Release Notes

• Also look at: http://userforums.opnet.com

–Info on ALL OPNET products

–Discussion Groups

Hi & i ktest

–Hints & tricks


2/10/2011

164


OPNET Training Courses

•OPNET provides Instructor-led training for all of our products. –OPNET Training Facilities;OPNET Training Facilities;

• Bethesda, Cary, Dallas, Nashua, Santa Clara

–On-site at YOUR facilities–On the Web via WebEx

•For AppTransaction Xpert :–2-day Instructor-led-training

test•Register for courses online at: www.opnet.com/Training


OPNET Certification Program

Become recognized as an expert user of OPNET products…

Certification covers relevant topics such as:

• Management / AdministrationManagement / Administration

• Configuration

• Product Selection

• Product Operation

• Networking Knowledge

Upon successful completion:

test

p p

• Letter

• Certificate

• Individual certification ID

• Valid for 2 years329 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc.

2/10/2011

165


AppTransaction Xpert Online Training

test


OPNET Web Pages - Training Resource

test


2/10/2011

166


OPNETWORK is the largest event of its kind, attracting thought leaders from industry, government, and academic communities from all over the world to

test

332

RSVP at: www.opnet.com/opnetwork2010

and academic communities from all over the world, to advance best practices for:

• Application Performance Management• Network Engineering, Operations, and Planning• Network R&D


Course Evaluation

We’ll use this information to improve our courses and teaching methods. Please enter as much comment material as you’d like… the more info you add, the better we’ll get!

Point your browser to:

http://eval.opnet.com

Please enter Instructor and Course info…– Course: Introduction to AppTransaction Xpert – Course Date:– Instructor Name:

FREE

Stuff !!test

Thank You in advance for your participation!


2/10/2011

167


test


Cover of ACE Analyst Lecturephucchau.tran.free.fr/book/AppTransaction Xpert_Lecture_16.0... ·...

Documents

Transcript of Cover of ACE Analyst Lecturephucchau.tran.free.fr/book/AppTransaction Xpert_Lecture_16.0... ·...