Automated Performance Profiling with Continuous Integration · 2014-01-20 · W11 Concurrent Class...
Transcript of Automated Performance Profiling with Continuous Integration · 2014-01-20 · W11 Concurrent Class...
W11 Concurrent Class
10/2/2013 1:45:00 PM
"Automated Performance
Profiling with Continuous
Integration"
Presented by:
Ivan Kreslin
Mitchell International Inc
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com
Ivan Kreslin
Mitchell International
Ivan Kreslin is a principal software development engineer at Mitchell International, North
America's leading provider of property and casualty claims technology solutions for most major
insurance providers and collision repair shops. Ivan leads the test automation development
efforts in the auto casualty solutions division, creating tools and architecting and integrating the
tools into automation frameworks and libraries shared among the different feature teams. Ivan
brings more than twenty-one years of software industry experience, most focused on quality and
automation.
Automated Performance Profiling 9/20/2013
1
Automated Performance Profiling in Continuous Integration
Ivan Kreslin Principal Software Development Engineer
Confidential and Proprietary. ©2013 Mitchell International, Inc.
Automated Performance Profiling
Overview • Historically performance tests
have been run too late
• Learn how to capture regressions in performance during Continuous Integration (CI) by using Microsoft’s Performance Profiling tools functionality integrated into a test automation framework.
– A test framework that enables existing and new test automation to be used for both functional testing and performance profiling purposes.
Automated Performance Profiling 9/20/2013
2
Confidential and Proprietary. ©2013 Mitchell International, Inc.
Automated Performance Profiling
Performance Profiling in Visual Studio • Before attempting to automate performance
profiling, try it out within Visual Studio.
Confidential and Proprietary. ©2013 Mitchell International, Inc.
Automated Performance Profiling
Performance Profiling w/Command Line • For test automation, the command line tools are used.
Automated Performance Profiling 9/20/2013
3
Confidential and Proprietary. ©2013 Mitchell International, Inc.
Automated Performance Profiling
Data Collection Method • Three main ways for collecting performance
profiling data:
– CPU Sampling
– Instrumentation (i.e. trace).
– Concurrency
• Our automation framework uses Instrumentation in order to obtain detailed timing data each time a function gets called.
Confidential and Proprietary. ©2013 Mitchell International, Inc.
Automated Performance Profiling
Binary Instrumentation • Performance Profiling requires
instrumented binaries.
• The Profiling Tools instrument binaries by injecting timing info at start and end of each function.
– Related symbol files are also required.
– Setting of Environment Variables on test machine:
• PATH, NT_SYMBOL_PATH
Automated Performance Profiling 9/20/2013
4
Confidential and Proprietary. ©2013 Mitchell International, Inc.
Automated Performance Profiling
Visual Studio Profiling (VSP) Data Files
• A vsp file is generated by the profiling tools after each test run.
– The vsp file collects function timing details.
• Vsp files can get relatively large, growing whenever functions are called.
– Running of our current tests result in files
ranging from 25MB to 500MB
• If changes are made to a test affecting the total performance then a new baseline needs to be captured for usage in any future test runs.
Confidential and Proprietary. ©2013 Mitchell International, Inc.
Automated Performance Profiling
Baseline Data • To determine if regressions in performance have
occurred we need a baseline to compare against.
– To capture baseline data: We run the test same way as we plan to do later in continuous integration.
• The baseline vsp gets stored in TFS and can then be used in then test runs.
Automated Performance Profiling 9/20/2013
5
Confidential and Proprietary. ©2013 Mitchell International, Inc.
Automated Performance Profiling
Determining Success of a Test Run
• Time spent within calls to instrumented binaries is compared against the baseline.
• If total time is greater than a specified % allowed by the test -> test will be logged as having failed.
– We chose this logic as some relatively high percent diffs can often be just fractions of a ms.
Confidential and Proprietary. ©2013 Mitchell International, Inc.
Automated Performance Profiling
Creating the Automated Tests • Automation Engineers create tests as they would
any functional test.
– All referencing a common Automation Framework.
• During test initialization, the tests config xml is checked to determine if performance profiling is to be used, the xml also specifies additional needed info such as:
– TFS branch location of code being tested
– List of binaries to be instrumented
– Path to baseline vsp file for the test…
Automated Performance Profiling 9/20/2013
6
Confidential and Proprietary. ©2013 Mitchell International, Inc.
Automated Performance Profiling
Running the Tests • Tests are run using the latest CI release build.
• Tests can be run like any other framework referencing Automated test.
– From Visual Studio, MTM, mstest.exe or Part of the build process as is the case with our tests
Confidential and Proprietary. ©2013 Mitchell International, Inc.
Automated Performance Profiling
Profiling Test Execution Overview
• Automation framework creates and executes 2 bat files used during by the process
(1) Binary instrumentation
(2) Runs Perf Profiling commands
& launches application/service
• Diff report is inserted into a sql table
• Timing Data is analyzed and success/fail is determined
• Build/Test completes and report is generated
Automated Performance Profiling 9/20/2013
7
Confidential and Proprietary. ©2013 Mitchell International, Inc.
Automated Performance Profiling
Profiling Tool Command Details
• 1st step instruments the binaries, e.g.
– VSInstr AnalyzeBridge.dll…
• 2nd step launches app and runs Perf tools, e.g.
– VSPerfCLREnv /TraceOn
– VSPerfCmd /Start:Trace /Output:<VSP>
• ASP.NET apps use VSPerfAspNetCmd.exe instead
– <Application.exe> ( or net start <ServiceName>)
– VSPerfCmd /shutdown
– VSPerfReport /output & /diff
Confidential and Proprietary. ©2013 Mitchell International, Inc.
Automated Performance Profiling
Test Environment
• State of machines/db’s… used to run profiling tests needs to be constant between test runs.
• Use Physical machines if VM’s are inconsistent
– Testing with VM’s showed performance varying up to 20% run to run vs. 2% for physical machines.
• Time to complete running test depends on:
– Amount & number of times functions are executed.
– Our libraries enable running a test multiple times to obtain a median time & reduce false positives.
Automated Performance Profiling 9/20/2013
8
Confidential and Proprietary. ©2013 Mitchell International, Inc.
Automated Performance Profiling
Reports • After a test run has completed a report is automatically
generated and sent out with details of the run.
– Report lists functions with the greatest difference vs. baseline along with recent TFS check-in info.
– Any regressions get assigned to appropriate developers
• Reports can be customized whichever
way teams wish/need them to be.
Confidential and Proprietary. ©2013 Mitchell International, Inc.
Automated Performance Profiling
Sample Report Table
Automated Performance Profiling 9/20/2013
9
Confidential and Proprietary. ©2013 Mitchell International, Inc.
Automated Performance Profiling
Steps To Get Started 1. Validate your application can be profiled
2. Integrate performance profiling tools execution and data/TFS handling into test automation framework
3. Create/Identify:
– Automated tests for profiling usage
– List of Application binaries to be instrumented
– Baseline vsp file(s).
– Test environment: machine/db… as needed
– TFS build definition used to build & deploy the
application + build & run the tests.
Confidential and Proprietary. ©2013 Mitchell International, Inc.
Automated Performance Profiling
Demo
Automated Performance Profiling 9/20/2013
10
Confidential and Proprietary. ©2013 Mitchell International, Inc.
Automated Performance Profiling
Questions? • Contact: [email protected]