Post on 05-Sep-2014
description
Debugging .NET Applications with WinDBG
Cory Foyhttp://coryfoy.com | @cory_foy
Agenda
.NET Fundamentals Debugging Crashes Debugging Memory
Leaks Debugging Hangs WinDBG Tips and
Tricks Other Tools
.NET Fundamentals
CLR Fundamentals Memory Management Basics Debugging Fundamentals
CLR Fundamentals
Managed Assembly Layout
PE Header
CLR Header
IL (Intermediate Language)
Metadata Header
Metadata
CLR Fundamentals
.NET Runtime in a Process
Demo: Viewing Managed Objects
Cory Foyhttp://coryfoy.com | @cory_foy
Viewing Managed Objects
Start up the app and click on the Managed Objects button
Start up WinDBG and attach to the executable (click “No” if it asks you to save the workspace) then load SOS by typing “.loadby sos mscorwks” and hitting enter. Also load symbols by typing “.symfix C:\symbols” and hitting enter, then typing “.reload” and hitting enter
Viewing Managed Objects
View All Threads by typing in “~” and hitting enter. View all managed threads by typing in “!threads” and hitting enter. Note there are 5 native threads, but only 2 managed threads. Also note that thread 2 is marked as (Finalizer) – that’s the thread responsible for finalization
Change to the main thread by typing “~0s” (tilde zero s). This tells WinDBG to switch to thread 0. Then type “!clrstack” to see where we are in the application. Note we are in the Main method right now.
Viewing Managed Objects
Next, let’s look at the objects on the heap. Type “!dumpheap –stat”. You’ll see many objects. You can filter by typing things like -min or –max as parameters to dumpheap.
Now find all of our objects on the heap by typing “!dumpheap –type WinDBG”. This filters for any object with the text “WinDBG” in it’s name. The top list are the actual objects and their memory locations, while the bottom list are the type names. You can figure out which is which by matching up the MethodTable addresses (MT Column)
Viewing Managed Objects
We can view a specific object using “!dumpobj” (or “!do” for short). We’ll look at the HelloWorld Object by typing “!do <address>” where address is from the !dumpheap command earlier. We can see we have a string field called “hello” and a world property.
We can then look at the string by typing “!do <address>”. In this example, the address would be 263a000. We can see that field contains the string “Hello”.
Viewing Managed Objects
When we did a !do on the HelloWorld object, one piece of information was the MethodTable address. We can use the !dumpmt command to see what methods the object exposes. Type “!dumpmt –md <methodtableaddress>”. If the JIT Column has “PreJIT” then the method came from an Ngen’d object. If it has JIT, then the method has been called and JIT’d. If it has NONE, the method hasn’t been called.
You can dissassemble IL that has been JIT’d by passing in the MethodDesc address to “!U”
Viewing Managed Objects
If we’ve attached to the process doing a live debug (which we’re doing here), then you can set breakpoints using !bpmd. For example, we can have it breakpoint just before the MessageBox shows up by passing in the assembly name and fully qualified type. You can use bl to see the breakpoints.
We then type “g” to release the current break we have on the app, and click the Managed Objects button again.
We’ll then see WinDBG hit the breakpoint. We can run !clrstack to see what led us to that call
CLR Fundamentals
Threads Managed Threads is an object – it lives
on native system threads CLR Threads▪ Finalizer▪ Garbage Collector (Server GC)▪ Debugger▪ Timer▪ Threadpool (I/O and Worker)
CLR Fundamentals
Just-In-Time compilation Program makes a call to a method The .NET Runtime checks to see if the
method has been called before If so, it executes the JIT’d code If not, it compiles the IL code and stores
it in memory, updating the MethodDesc
.NET Fundamentals
CLR FundamentalsMemory Management Basics Debugging Fundamentals
Memory Management
Stacks versus Heaps Stack – First in / First Out Heap – access by address
Garbage Collector Heap Where all objects are stored Broken into 3 generations and one Large
Object Heap Large Object > 85,000 bytes
Memory Management
Garbage Collector Sequence Suspend Execution Engine Mark objects without roots Plan (budgets, fragmentation) Sweep (delete market objects) Compact (move leftover objects to back
of heap) Restart Execution Engine
Memory Management
Memory Leaks Possible in both Managed and
Unmanaged Code Use Perfmon to check for symptoms Unmanaged Leak▪ Private Bytes Increase, #Bytes In All Heaps
stays flat Managed Leak▪ Both Private Bytes and Bytes In All Heaps
increase▪ Need multiple dump files
Memory Management
http://blogs.msdn.com/tess/archive/2008/03/17/net-debugging-demos-lab-6-memory-leak-review.aspx
Memory Management
Why do we leak managed memory? Objects not being released “Pinned” memory Finalized Objects (Destructors on
Managed Objects)▪ Finalized objects require an additional GC
cycle to be cleaned since they have to go in the finalizer thread to run
Memory Management
Exception Handling Workflow Exception Occurs (create an exception
object) Notify Debugger (1st Chance Exception) Look for a handler by walking up the call
stack If handler found, let it handle exception If not, throw a 2nd Chance Exception and
terminate the process
.NET Fundamentals
CLR Fundamentals Memory Management BasicsDebugging Fundamentals
Debugging Fundamentals
Typical Problems in Production System hangs or deadlocks Fatal Exceptions Data Loss or inconsistency Performance Problems Excessive Memory Usage App Pool Restarts (slow access to
ASP.NET pages)
Debugging Fundamentals
Approaching the problem Be the application (visualize what
could cause the problem) Use Application and System Logs Try to reproduce in Development or
Staging environments Create a hypothesis and use
WinDBG to validate
Debugging Fundamentals
Debugging Production Apps Typically don’t have Visual Studio
installed, or access to the remote debugger
Attaching a debugger freezes all threads
Capture memory dumps▪ At time of crash / exception▪ Over time to troubleshoot hangs / leaks
Agenda
.NET FundamentalsDebugging
Crashes Debugging Memory
Leaks Debugging Hangs WinDBG Tips and
Tricks Other Tools
Debugging Crashes
Application seems to work fine, but something happens Unhandled Exception Dialog App “disappears”
Steps Desktop App: Attach and stop on
Exceptions Web App: Use DebugDiag to capture
memory dump
Demo: Debugging Crashes
Cory Foyhttp://coryfoy.com | @cory_foy
Debugging Crashes
Open Application and click the “Crash” button”. Unhandled Dialog should appear and kill the app. Start app back up and attach to it with WinDBG, then load SOS. Once SOS is loaded, type “sxe clr” to tell WinDBG to break on all .NET Exceptions, then type “g” and click on the Crash button again.
Because we’ve enabled exception notification, WinDBG breaks on the first-chance exception.
Debugging Crashes
The first place to look is the exception object itself. We can either do a “!do” on the object address listed, or simply type “!pe”. You can also pass an address to !pe if you need to view other than the last exception thrown.
We can see there is a file not found exception. Let’s see where we were at.
So it looks like the app calls “ReadAllText” from a method called “Crash_Click” from the Form1 object. If we have access to Source, we’d start there. If not, we can find the method address and dissassemble
Debugging Crashes
Note that if we click “g” at this point, WinDBG breaks again. This is because the exception we first saw was *not* the crash reason. Like the previous exception, this one is listed as a First Chance Exception.
If we click “g” again, we’ll see WinDBG breakpoint again. Note that it clearly tells us this is a second chance exception – the next thing that will happen is process termination.
Demo: Debugging Crashes on Application Startup
Cory Foyhttp://coryfoy.com | @cory_foy
Debugging Crashes On Startup
Open the App and check the “Crash on Startup” option, then close the app and restart. Notice it immediately crashes.
The challenge is that we won’t have time to attach a debugger. You can set registry keys to automatically launch a debugger, or we could launch the app from the debugger, which we’ll do here.
Start up WinDBG and go to File->Open Executable. Browse to the WinDBGDemo executable and select it.
Debugging Crashes On Startup
At this point, we are in the PE load of the application, meaning it hasn’t even decided if this is a native or managed app yet. Which also means no .NET libraries are loaded, so we can’t load SOS.
What we can do is set a breakpoint to fire when the .NET runtime is loaded, at which point we can set the handlers we need. Type “sxe ld mscorwks” which means set WinDBG to break when mscorwks is loaded. We can now load SOS and set the CLR Exception breakpoints. Now type “g” till our app breaks. You can now debug it as a crash.
Debugging Crashes On Startup
If you have a Windows Service which is crashing on startup, then you’ll need to modify some registry keys and permissions on the service. For more information, see the following KB article, or the blog post from our site
(Note: When you finish this demo, go to your user directory\AppData\Local\Microsoft\WinDBGDemo.exe\1.0.0.0 and modify the user.config file to have CrashOnStartup to be False)
http://support.microsoft.com/kb/824344
http://blog.coryfoy.com/2008/06/debugging-a-net-windows-service-that-is-crashing-on-startup/
Agenda
.NET Fundamentals Debugging CrashesDebugging
Memory Leaks Debugging Hangs WinDBG Tips and
Tricks Other Tools
Debugging Memory Leaks
Memory Usage Continues to grow in the app
May cause app pool restarts in ASP.NET
Out of Memory exceptions when you still have memory free
Several Reasons Unreleased objects “Pinned” memory You have plenty of memory! (GC only
runs when it feels memory pressure)
Debugging Memory Leaks
Diagnosing Memory Leaks PerfMon to determine if it is a managed
leak Take multiple memory dumps to
determine what is happening For the objects on the heap, find out why
they are sticking around Check the GC Generation for the objects
Demo: Debugging Memory Leaks
Cory Foyhttp://coryfoy.com | @cory_foy
Debugging Memory Leaks
Start the app and task manager, then click on the Memory Leak button. Note that the memory is still increasing.
Fire up Perfmon to see if this is a managed or native leak. Start->Run->Perfmon. You can remove the Processor Time counter and add the following counters: .NET CLR Memory -> #Bytes in all Heaps -> WinDBGDemo and Process -> Private Bytes -> WinDBGDemo.
You should see that both are growing at about the same rate indicating a managed leak
Debugging Memory Leaks
To debug this, we’ll need to take two snapshots of the application far enough apart to see memory difference. You can do this right from Task Manager. Create two, about 15 seconds apart. You can now kill the application.
Open two instances of WinDBG and open each memory dump file you created in a different instance of WinDBG. Load up SOS in each one as well.
Debugging Memory Leaks
What we’re wanting to do is compare the two memory dumps to see if we can spot the culprit. Using !VMStat on each file we can see that memory usage is certainly increasing.
And if we do a “!dumpheap –stat” on each one, we can see something quite startlingly – 230,000 additional objects are on the heap!
Debugging Memory Leaks
Normally you’d have to dig through the heap to find which objects increased, but we see something interesting. An object called “LeakForm” has had 900 instances created. Perhaps those aren’t being cleaned up?
Let’s look at one. We’ll see all the instances by doing a “!dumpheap -type WinDBGDemo.LeakForm” then picking any object at random to do a !do on
With that object address, let’s see what is holding on to it with “!GCRoot <address>”
Debugging Memory Leaks
We see that the LeakForm is being held on to an object, which is of type System.Object[]. If we do a !do on it, we don’t see much of interest
But if we compare the arrays between the first dump and the second dump using the “!da <address>” command, we find a clue. Between the first and second memory dumps, all of the extra LeakForms were added to this array!
At this point, we examine the source code to see what is going on.
Agenda
.NET Fundamentals Debugging Crashes Debugging Memory
LeaksDebugging
Hangs WinDBG Tips and
Tricks Other Tools
Debugging Hangs
Two types of hangs Low CPU Hang▪ Generally caused by a deadlock
High CPU Hang▪ App in a tight loop, or under heavy load
Debugging Hangs
Diagnosing Hangs High CPU Hang▪ Multiple memory dumps▪ Compare which thread is getting processor
time Low CPU Hang▪ Walk through the waiting threads and match
locks▪ Very tedious process
Demo: Debugging High CPU Hangs
Cory Foyhttp://coryfoy.com | @cory_foy
Debugging High CPU Hangs
Start the app and Task Manager. Click on the High CPU Hang. You should see a single processor spike (100% on 1CPU). In this example, I have a 4CPU system, so a CPU of 25% indicates it is fully utilizing one processor.
Take two memory dumps from Task Manager about 15 seconds apart. You can then kill the application.
Debugging High CPU Hangs
Open the two memory dump files in different WinDBG instances and load SOS.
The first thing we need to know is if a single thread is using up all of the time, indicating a possible loop. We can use the command !runaway to see which threads are getting CPU time
We can see from this that thread 0 was using up all of the time in between the two memory dumps
Debugging High CPU Hangs
So what is thread 0 doing? By running !clrstack on each memory dump, we can see that the call stacks are the same, possibly indicating that this is the culprit.
I say possibly because if a bunch of calls are happening to different methods, we very well could have gotten “lucky” and seen the same method. Always verify your assumptions.
In this case, looking at the source (either real source, or through reflector) we see the following. I think we have our culprit
Agenda
.NET Fundamentals Debugging Crashes Debugging Memory
Leaks Debugging HangsWinDBG Tips and
Tricks Other Tools
Demo: WinDBG Tips and Tricks
Cory Foyhttp://coryfoy.com | @cory_foy
WinDBG Tips and Tricks
You can get quite a bit of help with both WinDBG and SOS.
For the WinDBG help content, type “.hh” and it will open the WinDBG CHM file. You can also type “.hh <command>” and it will take you to that section
For SOS, you can type “!help” to see all available commands, or “!help <command>” to see a specific one
WinDBG Tips and Tricks
You can use the shell command to run the output of a WinDBG command to an external process and the results are displayed back into the UI. For example, to quickly find the shared domain, you can run ‘.shell –ci “!dumpdomain” find /i “shared domain”’
You can also take advantage of this to do integrations with PowerShell
WinDBG Tips and Tricks
When viewing stacks, you can use !clrstack to view the managed stack. You can also use the command “k” to view the native stack
You can also output the stack for all threads by combining the “~” command with a star (for all threads) and the command. For example “~*k” or “~*e!clrstack” (which means execute the command !clrstack)
WinDBG Tips and Tricks
You can do a lot with automation of WinDBG. For example, you can have WinDBG execute commands when certain breakpoints are hit
You can also automate what happens when CLR Exceptions are hit by using the –c option with sxe. In this example, every time we hit an exception we would spit out the exception and CLR stack trace.
WinDBG Tips and Tricks
You can also loop through various elements. For example, if we wanted to dump out all strings greater than 5k that were on the heap, we could do the following with the “.foreach” command
Finally, if you want to adjust the colors, you can do that in View->Options
Agenda
.NET Fundamentals Debugging Crashes Debugging Memory
Leaks Debugging Hangs WinDBG Tips and
TricksOther Tools
Additional Tools
DebugDiag Used to automate the capture of
exceptions for IIS processes ADPlus
Used to capture dumps from command line
Managed Debuggers (CorDbg, MDbg) Managed Debugging Assistants
(Visual Studio) Profilers
More Information
http://blogs.msdn.com/tess microsoft.com/whdc/DevTools/
Debugging http://windbg.info http://www.coryfoy.com foyc at cory foy dot com @cory_foy on Twitter Slides will be posted on CoryFoy.com