Post on 25-Dec-2015
Power Debugging
Kapil VaswaniResearcherRigorous Software EngineeringMicrosoft Research India
FT54
Sandeep KaranthRSDE
Advanced Development and Prototyping Microsoft Research India
Sriram Rajamani, Aditya Nori (Rigorous Software Engineering, MSRI)Joseph Joy, B. Ashok, Gopal Srinivasa (Advanced Development and Prototyping, MSRI)Hongkang Liang, Vipindeep Vangala (Windows Sustained Engineering)Trishul Chilimbi (Runtime Analysis and Design, MSR)Abhik Roychoudhury (National University of Singapore)Ben Liblit (University of Wisconsin)
Debugging is hard!
Symptom of bug
Root cause
ManualTediousTime
consuming
Ask experts
Check bug database
Check version history
Reproduce bug
Trace in a debugger
Debugging Yesterday, Today and Tomorrow
How else can we help diagnose failures?+Visual Studio
IntellitraceTM
+Visual StudioTest Elements
+Visual Studio Test Impact Analysis
Power Debugging
Holmes Statistical debugging toolUse large test suites to diagnose failures
Debug AdvisorRecommendation system for bugsMines software repositories for information related to a bug
DarwinTool for debugging regressionsUse a previous, stable version of an application to diagnose failures
HolmesStatistical Debugging
Kapil Vaswani, Aditya Nori (Rigorous Software Engineering, MSRI)Sandeep Karanth (Advanced Development and Prototyping, MSRI)Trishul Chilimbi (Runtime Analysis and Design, MSR)Ben Liblit (University of Wisconsin)
HolmesWhere testing meets debugging
> Programs are often put through rigorous testing > Large test suites> Many passing tests, some failing tests
> Can test suites help us find the cause of failures?
Statistical Debugging with Holmes
> Collect profiles/coverage data from a large number of (successful and failing) test cases
> Look for code paths that strongly correlate with failure
Debugging with Holmes
Test suiteAutomated/ManualVisual Studio unit testsVisual Studio Test Elements
Code coverageHolmes path coverageHistorical debugging
0111011..
Test resultsPass/fail
HolmesStatistical analysis
Potentialroot causes
HolmesVisual Studio Integration
Sandeep KaranthRSDEAdvanced Development and Prototyping
demo
Holmes available for download today!
http://research.microsoft.com/holmes Try it and give us feedback!
announcing
Debugging with Holmes
Test suiteAutomated/manualVisual Studio unit testsVisual Studio Test Elements
Code coverage
0111011..
Test resultsPass/fail
HolmesStatistical analysis
Potentialroot causes
HolmesStatistical analysis
Code coverage
Code coverage for Holmes
> Statement/block/arc coverage insufficient
> Path coverage> Track acyclic, intra-procedural
path fragments> Why paths?
> Paths represent scenarios> Bugs often occur in complex
scenarios> Profile with low overheads
(roughly 10 – 30%)
a
b
c d
e
f
Statistical Analysis
> Measuring correlation straight-forward> But there is a pitfall!
> cum hoc ergo propter hoc, a logical fallacy
> Correlation does not imply causality> Examples
> Exception handling code> Error recovery routines
Cause and correlation
> An analysis that distinguishes cause from correlation> Context of the path - method/loop/try-
catch block containing the path> Look for paths that strongly correlate with
failure> But context does not correlate with
failure> Very effective in practice!
Statistical Analysis
> Inputs to analysis> Path coverage for each test case> Outcome of each test case
> Compute four statistics for each path
number of successful test cases in which p is covered
number of failing test cases in which p is covered
number of successful tests in which the context of p was covered
number of failing tests in which the context of p was covered
Statistical Analysis
𝒊𝒏𝒄𝒓𝒆𝒂𝒔𝒆 ( p )=Fcov ( p )
Scov ( p ) +Fcov ( p ) −contex t ( p)
𝒓𝒆𝒄𝒂𝒍𝒍 ( p )=Fcov ( p )
F total
𝒄𝒐𝒏𝒇𝒊𝒅𝒆𝒏𝒄𝒆 ( p )=mean(increase ( p ) , recall (p ))
Context: How much is the context of a path correlated with failure?
Increase: How much more is the path correlated with failure?
Recall: What fraction of all failures occur when this path is covered?
Confidence: Overall measure that combines increase and recall
𝒄𝒐𝒏𝒕𝒆𝒙𝒕 (p )=Fcontext ( p )
Scontext ( p ) +Fcontext ( p )
Giving Holmes a hand
> Write more test cases!> Use automated test generation tools like
Pex> How many?
> Typically, 10-20 related failing tests with ~100 passing tests suffice
>>FUTURE
Summary
> Holmes available on http://research.microsoft.com/holmes > Ships with a tool for measuring path
coverage> Integrates with Visual Studio and Test
Elements> Supports managed code> Supports automated and manual tests
> Ongoing work> Support for unmanaged code> Support for historical debugging traces
> Try it and give us feedback!
DarwinAutomatically Root-causing RegressionsKapil Vaswani (Rigorous Software Engineering, MSRI)Abhik Roychoudhury (National University of Singapore)
Regressions
> Changes that break functionality> Often uncovered by regression testing
> Debug by comparing buggy version with previous, correct version> Doesn’t work when too many changes> Doesn’t work for unmasking regressions
Debugging by comparing test cases> Compare trace of the
failing test case with a similar, passing test case
> Problem – such test cases don’t exist!
Root cause
Darwin – Key ideas
> Define notions of similarity between test cases
> Automatically generate similar, passing test given a failing test case
Test Similarity
> Given> Two versions of an
application P and P’> A test T that passes
on P and fails on P’
> Similarity> A test T’ is similar to T
if T’ and T follow the same control flow path in P but different paths in P’
Old version
New version
Root cause
Test Generation in Darwin
> Problem of finding similar tests as a constraint solving problem> Using techniques similar to Pex
Darwin at work
void dodash(char delim, char* src, int* i, char* dest, int* j, int maxset)
{int k;bool junk;char escjunk;
while ((src[*i] != delim) &&
(src[*i] != ENDSTR) {
if (src[*i] == ESCAPE) {
escjunk = esc(src, i);
junk = addsrt(escjunk,
dest, j, maxset);}else {
…}
}
void dodash(char delim, char* src, int* i, char* dest, int* j, int maxset)
{int k;bool junk;char escjunk;
while ((src[*i] != delim) &&
(src[*i] != ENDSTR) {
if (src[*i - 1] == ESCAPE) {
escjunk = esc(src, i);
junk = addsrt(escjunk,
dest, j, maxset);}else {
…}
}
% [0-9][^9-B][@t][^a-c]
% [0-9][^9-B][00][^a-c]
Failing test case
Passing test case generated by Darwin
Old New
>>FUTURE
Current status
> Prototype based on Pex> Automatically root caused regressions
in large applications> Web servers (HTML pages)> Image processing applications (jpeg
images)
> Working on VS integration, supporting multi-threading, …
> Watch this space! http://research.microsoft.com/darwin
DebugAdvisorA recommendation system for bugs
Sriram Rajamani (Rigorous Software Engineering, MSRI)Joseph Joy, B. Ashok, Gopal Srinivasa (Advanced Development and Prototyping, MSRI)Hongkang Liang, Vipindeep Vangala (Windows Sustained Engineering)
A Common Scenario
Tester/developerreceives bug report
Has this or similar bug been
looked at or fixed before?
What do we know about this kind of
bugs?
Who should I ask for help?
Where should I start
looking?
What You KnowThe customer experiences some deadlocks on a server. The problem is random and may occur from several times a week to once a month. The system looks hung because the global resource 'ObpInitKillMutant' is help by a thread which tries to close a file forever. So all the processes having a thread waiting on 'ObpInitKillMutant' stop working fine. Drivers such as TCP/IP continue to respond normally but it's impossible to connect to any share.
0: kd> !thread 82807020ChildEBP RetAddr Args to Child80c7a028 00000000 00000000 ntkrnlmp!IopAcquireFileObjectLock+0x5882a6d7a0 80c7a028 00120089 ntkrnlmp!IopCloseFile+0x7982a6d7a0 80c7a010 80f6da40 ntkrnlmp!ObpDecrementHandleCount+0x11200000324 7ffdef01 00000000 ntkrnlmp!NtClose+0x17000000324 7ffdef01 00000000 ntkrnlmp!KiSystemService+0xc900000324 80159796 000000c9 ntkrnlmp!ZwClose+0xb000000c9 e185f648 00000000 ntkrnlmp!ObDestroyHandleProcedure+0xd809e3008 801388e4 82a6d926 ntkrnlmp!ExDestroyHandleTable+0x4800000001 82a6d7a0 7ffde000 ntkrnlmp!ObKillProcess+0x4400000001 82a6d7a0 82a6d7f0 ntkrnlmp!PspExitProcess+0x5400000000 f0941f04 0012fa70 ntkrnlmp!PspExitThread+0x447ffffffff 00000000 00002a60 ntkrnlmp!NtTerminateProcess+0x13cffffffff 00000000 00002a60 ntkrnlmp!KiSystemService+0xc900000000 00000000 00000000 NTDLL!NtTerminateProcess+0xb
REGISTERS:eax=00000005 ebx=e3185488 ecx=0000083c edx=e2dddc68
Textual descriptionof bug
Stack trace
Processor state
Debug Advisor Search
Similar Bugs
Related Information
Debug Logs
Query processing
QueryStack traceCode snippetsEmails
Feature parsersStack trace parserRegister information parser
Query engin
e
RepositoriesVersion controlBug repository
Debug logs
Similar bugs
Bug repository
Relationship builder
Relationship graph
Link Analysis
Debugadvisorreport
Deployment feedback
> Deployed internally for over 6 months> Used by several developers
77%
23%
Unsolicited feedback129 users, 628
queries
Useful Not useful
75%
25%
Solicited feedback20 bugs
Useful Not useful
>>FUTURE
Summary and Status
> Much better precision and recall compared to full text search
> Effectiveness depends on access to large repositories> Evaluating effectiveness with smaller,
more representative repositories> Exploring potential integration with Visual
Studio
Summary
> Debugging is getting harder!> Debugging tools need to evolve> Actively help diagnose failures
> Three tools that assist/automate debugging> Exploit by-products of a typical software
lifecycle (tests, versions, repositories)> Holmes available for download, others
will follow
Related talksCode Visualization, UML, and DSLs Cameron Skinner Tuesday 4:30 PM
Extending the Microsoft Visual Studio 2010 Code Editor to Visualize Runtime Intelligence
Gabriel Torok & Bill Leach Wednesday 4:30 PM
A Lap Around Microsoft Visual Studio and Team Foundation Server 2010
Cameron Skinner & Mario Rodriguez Thursday 10:00 AM
Microsoft Visual Studio Lab Management to the Build Setup Rescue Vinod Malhotra Thursday 10:00 AM
Scrum in the Enterprise and Process Customization with Microsoft Visual Studio 2010
Simon Bennett & Stuart Preston Thursday 1:45 PM
Advanced Diagnostics, IntelliTrace™ and Test Automation Habib Heydarian Thursday 1:45 PM
Power Tools for DebuggingKapil Vaswani & Sandeep Karanth Thursday 3:00 PM
Automating "Done Done" in the Team Workflows with Microsoft Visual Studio Ultimate and Team Foundation Server 2010
Jamie Cool & Brian Randell Thursday 3:00 PM
Resources
> Holmes> Download: http://research.microsoft.com/holmes> Forum: http://blogs.msdn.com/holmes
> Technical papers> Holmes: Effective Statistical Debugging via Efficient Path
Profiling, Trishul Chilimbi, Ben Liblit, Krishna Mehra, Aditya Nori and Kapil Vaswani, ICSE 2009
> Darwin: An Approach for Debugging Evolving Programs, Dawei Qi, Abhik Roychoudhury, Zengkai Liang and Kapil Vaswani, FSE 2009
> DebugAdvisor: A Recommender System for Debugging, B. Ashok, Joseph Joy, Hongkang Liang, Sriram Rajamani, Gopal Srinivasa, and Vipindeep Vangala, FSE 2009
YOUR FEEDBACK IS IMPORTANT TO US!
Please fill out session evaluation
forms online atMicrosoftPDC.com
Learn More On Channel 9
> Expand your PDC experience through Channel 9
> Explore videos, hands-on labs, sample code and demos through the new Channel 9 training courses
channel9.msdn.com/learnBuilt by Developers for Developers….
© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.