Windows Memory Dump Analysis - Software Diagnostics Services
Windows Memory Dump Analysis - Software Diagnostics …...Windows Memory Dump Analysis . Dmitry...
Transcript of Windows Memory Dump Analysis - Software Diagnostics …...Windows Memory Dump Analysis . Dmitry...
-
Windows Memory Dump Analysis
Dmitry Vostokov Software Diagnostics Services
Version 4.0
-
Prerequisites Basic Windows troubleshooting
WinDbg Commands We use these boxes to introduce WinDbg commands used in practice exercises
© 2016 Software Diagnostics Services
-
Training Goals
Review fundamentals Learn how to analyze process dumps Learn how to analyze kernel dumps Learn how to analyze complete
(physical) and active dumps
© 2016 Software Diagnostics Services
-
Training Principles Talk only about what I can show Lots of pictures Lots of examples Original content and examples
© 2016 Software Diagnostics Services
-
Coverage Windows Vista, 7, 8, 10 Both x86 and x64 platforms Process, Kernel, Complete (Physical),
and Active memory dumps, Minidumps Crashes, Hangs, Memory Leaks,
CPU Spikes, Blue Screens (BSOD)
© 2016 Software Diagnostics Services
The main set of exercises is focused on Windows 10 x64 platform. All main exercises have their x86 equivalents from older Windows versions for additional practice.
-
Main Schedule Summary Day 1 Analysis Fundamentals (30 minutes) Process Memory Dumps (2 hours) Day 2 Process Memory Dumps (2 hours)
Day 3 Kernel Memory Dumps (2 hours) Day 4 Complete and Active Memory Dumps (2 hours)
© 2016 Software Diagnostics Services
Windows 10 and 8.1 x64 memory dumps
-
Optional Schedule Summary Day 1 Legacy Process Memory Dumps (2 hours) Day 2 Legacy Process Memory Dumps (2 hours) Day 3 Legacy Kernel Memory Dumps (2 hours)
Day 4 Legacy Complete Memory Dumps (2 hours)
© 2016 Software Diagnostics Services
Windows Vista and 7 x86 memory dumps
-
Part 1: Fundamentals
© 2016 Software Diagnostics Services
-
Process Space (x86)
Kernel Space
User Space
FFFFFFFF
800000007FFFFFFF
00000000
© 2016 Software Diagnostics Services
-
Process Space (x64)
© 2016 Software Diagnostics Services
Kernel Space
User Space
FFFFFFFF`FFFFFFFF
FFFF8000`00000000000007FF`FFFFFFFF
00000000`00000000
-
Application/Process/Module
© 2016 Software Diagnostics Services
Kernel Space
User Space (PID 102)
FFFFFFFF
800000007FFFFFFF
00000000
Notepad.exe
Notepaduser32.dll
user32
-
OS Kernel/Driver/Module
© 2016 Software Diagnostics Services
Kernel Space
User Space
FFFFFFFF
800000007FFFFFFF
00000000
Driver.sys
DriverNtoskrnl.exe
nt
-
Process Virtual Space
00000000 ... FFFFFFFF
© 2016 Software Diagnostics Services
User Space (PID 102)
FFFFFFFF
800000007FFFFFFF
00000000
Notepad
user32
Kernel Space
Driver
nt
-
Process Memory Dump
WinDbg Commands lmv command lists modules and their description
© 2016 Software Diagnostics Services
User Space (PID 102)
FFFFFFFF
800000007FFFFFFF
00000000
Notepad
user32
Notepad.exe.102.dmp
Kernel Space
Driver
nt
-
Kernel Memory Dump
WinDbg Commands lmv command lists modules and their description
© 2016 Software Diagnostics Services
User Space (PID 102)
FFFFFFFF
800000007FFFFFFF
00000000
Notepad
user32
MEMORY.DMPKernel Space
Driver
nt
-
Complete Memory Dump WinDbg Commands .process switches between process virtual spaces (kernel space part remains the same)
© 2016 Software Diagnostics Services
Kernel Space
FFFFFFFF
800000007FFFFFFF
00000000
Driver
nt
MEMORY.DMP
User Space (PID 102)
Notepad
user32
User Space (PID 204)
Calc
user32
-
Process Threads
WinDbg Commands Process dumps: ~s switches between threads Kernel/Complete dumps: ~s switches between processors .thread switches between threads
© 2016 Software Diagnostics Services
User Space (PID 306)
ApplicationA
user32
ntdll
Kernel Space
Driver
nt
TID 204TID
102
-
System Threads
WinDbg Commands Kernel/Complete dumps: ~s switches between processors .thread switches between threads
© 2016 Software Diagnostics Services
Kernel Space
Driver
nt
TID 306
User Space (PID 306)
ApplicationA
user32
ntdll
-
Thread Stack Raw Data
WinDbg Commands Process dumps: !teb Kernel dumps: !thread Complete dumps: !teb for user space !thread for kernel space Data: dc / dps / dpp / dpa / dpu
© 2016 Software Diagnostics Services
User Space (PID 306)
ApplicationA
user32
ntdll
Kernel Space
Driver
nt
TID 204
TID 102
Kernel Stack for TID 102
Kernel Stack for TID 204
User Stack for TID 204
User Stack for TID 102
-
Thread Stack Trace WinDbg Commands 0:000> k Module!FunctionD Module!FunctionC+130 Module!FunctionB+220 Module!FunctionA+110
User Stack for TID 102
Module!FunctionA
Module!FunctionB
Module!FunctionC
Saves return address Module!FunctionA+110
Saves return address Module!FunctionB+220
Module!FunctionD
Saves return address Module!FunctionC+130
Resumes from address Module!FunctionA+110
Resumes from address Module!FunctionB+220
Resumes from address Module!FunctionC+130
FunctionA(){ ... FunctionB(); ...}
FunctionB(){ ... FunctionC(); ...}
FunctionC(){ ... FunctionD(); ...}
Return address Module!FunctionC+130
Return address Module!FunctionB+220
Return address Module!FunctionA+110
© 2016 Software Diagnostics Services
-
Thread Stack Trace (no PDB)
WinDbg Commands 0:000> k Module+0 Module+43130 Module+32220 Module+22110
User Stack for TID 102
Module+22000
Module+32000
Module+43000
Saves return address Module+22110
Saves return address Module+32220
Module+54000
Saves return address Module+43130
Resumes from address Module+22110
Resumes from address Module+32220
Resumes from address Module+43130
FunctionA(){ ... FunctionB(); ...}
FunctionB(){ ... FunctionC(); ...}
FunctionC(){ ... FunctionD(); ...}
Return address Module+43130
Return address Module+32220
Return address Module+22110
No symbols for Module
Symbol file Module.pdb
FunctionA 22000 - 23000FunctionB 32000 - 33000FunctionC 43000 – 44000FunctionD 54000 - 55000
© 2016 Software Diagnostics Services
-
Exceptions (Access Violation)
WinDbg Commands address=???????? Set exception context (process dump): .cxr Set trap context (kernel/complete dump): .trap Check address: !pte
© 2016 Software Diagnostics Services
User Space (PID 306)
User Space (PID 306)
ApplicationA
user32
ntdll
ModuleA
TID 204
User Stack for TID 102
User Stack for TID 204
TID 102
Minvalid memory access
M00000000NULL pointer
-
Exceptions (Runtime)
© 2016 Software Diagnostics Services
User Space (PID 306)
User Space (PID 306)
ApplicationA
user32
ntdll
ModuleA
TID 204
User Stack for TID 102
User Stack for TID 204
TID 102
M throws error
-
Pattern-Oriented Diagnostic Analysis
Information Collection (Scripts)
Information Extraction (Checklists)
Problem Identification (Patterns)
Problem Resolution
Troubleshooting Suggestions
Debugging Strategy
Checklist: http://www.dumpanalysis.org/windows-memory-analysis-checklist Patterns: http://www.dumpanalysis.org/blog/index.php/crash-dump-analysis-patterns/
© 2016 Software Diagnostics Services
Diagnostic Pattern: a common recurrent identifiable problem together with a set of recommendations and possible solutions to apply in a specific context.
Diagnostic Analysis Pattern: a common recurrent analysis technique and method of diagnostic pattern identification in a specific context.
Diagnostic Problem: a set of indicators (symptoms, signs) describing a problem.
Diagnostics Pattern Language: common names of diagnostic and diagnostic analysis patterns. The same language for any operating system: Windows, Mac OS X, Linux, ...
http://www.dumpanalysis.org/windows-memory-analysis-checklisthttp://www.dumpanalysis.org/blog/index.php/crash-dump-analysis-patterns/http://www.dumpanalysis.org/blog/index.php/crash-dump-analysis-patterns/http://www.dumpanalysis.org/blog/index.php/crash-dump-analysis-patterns/http://www.dumpanalysis.org/blog/index.php/crash-dump-analysis-patterns/http://www.dumpanalysis.org/blog/index.php/crash-dump-analysis-patterns/http://www.dumpanalysis.org/blog/index.php/crash-dump-analysis-patterns/http://www.dumpanalysis.org/blog/index.php/crash-dump-analysis-patterns/
-
Part 2: Practice Exercises
© 2016 Software Diagnostics Services
-
Links Memory Dumps:
NOT IN THE PUBLIC PREVIEW VERSION
Exercise Transcripts: NOT IN THE PUBLIC PREVIEW VERSION
© 2016 Software Diagnostics Services
-
Exercise 0 Goal: Install Debugging Tools for Windows and learn how to
set up symbols correctly
Patterns: Incorrect Stack Trace
\AWMDA-Dumps\Exercise-0-Download-Setup-WinDbg.pdf
\AWMDA-Dumps\Exercise-Legacy.0-Download-Setup-WinDbg.pdf
© 2016 Software Diagnostics Services
-
Process Memory Dumps
Exercises P1 – P17
© 2016 Software Diagnostics Services
-
Exercise P1 Goal: Learn how to see dump file type and version, get a
stack trace, check its correctness, perform default analysis, list modules, check their version information, check process environment
Patterns: Manual Dump; Stack Trace; Not My Version; Environment Hint
\AWMDA-Dumps\Exercise-P1-Analysis-normal-process-dump-notepad-32.pdf
\AWMDA-Dumps\Exercise-Legacy.P1-Analysis-normal-
process-dump-notepad-32.pdf
© 2016 Software Diagnostics Services
-
Exercise P2 Goal: Learn how to list stack traces, check their correctness,
perform default analysis, list modules, check their version information, check process environment; dump module data
Patterns: Manual Dump; Stack Trace; Not My Version; Environment Hint; Unknown Component
\AWMDA-Dumps\Exercise-P2-Analysis-normal-process-dump-notepad-64.pdf
\AWMDA-Dumps\Exercise-Legacy.P2-Analysis-normal-
process-dump-notepad-64.pdf
© 2016 Software Diagnostics Services
-
Exercise P3 Goal: Learn how to list stack traces, check their correctness,
perform default analysis, list modules, check their version information, check thread age and CPU consumption
Patterns: Stack Trace Collection
\AWMDA-Dumps\Exercise-P3-Analysis-normal-process-dump-MicrosoftEdge-64.pdf
\AWMDA-Dumps\Exercise-Legacy.P3-Analysis-normal-
process-dump-iexplore-32.pdf
© 2016 Software Diagnostics Services
-
Exercise P4 Goal: Learn to recognize exceptions in process memory
dumps and get their context
Patterns: Exception Thread; Multiple Exceptions; NULL Pointer
\AWMDA-Dumps\Exercise-P4-Analysis-process-dump-ApplicationK-64-no-symbols.pdf
\AWMDA-Dumps\Exercise-Legacy.P4-Analysis-process-
dump-ApplicationK-32-no-symbols.pdf
© 2016 Software Diagnostics Services
-
Exercise P5 Goal: Learn how to load application symbols, recognize
exceptions in process memory dumps and get their context
Patterns: Exception Thread; Multiple Exceptions; NULL Pointer
\AWMDA-Dumps\Exercise-P5-Analysis-process-dump-ApplicationK-64-with-symbols.pdf
\AWMDA-Dumps\Exercise-Legacy.P5-Analysis-process-
dump-ApplicationK-32-with-symbols.pdf
© 2016 Software Diagnostics Services
-
Exercise P6 Goal: Learn how to recognize heap corruption
Patterns: Exception Thread; Dynamic Memory Corruption
\AWMDA-Dumps\Exercise-P6-Analysis-process-dump-
ApplicationL-32.pdf \AWMDA-Dumps\Exercise-Legacy.P6-Analysis-process-
dump-ApplicationL-32.pdf
© 2016 Software Diagnostics Services
-
Exercise P7 Goal: Learn how to recognize heap corruption and check
error and status codes
Patterns: Exception Thread; Dynamic Memory Corruption
\AWMDA-Dumps\Exercise-P7-Analysis-process-dump-ApplicationL-64.pdf
\AWMDA-Dumps\Exercise-Legacy.P7-Analysis-process-
dump-ApplicationL-64.pdf
© 2016 Software Diagnostics Services
-
Exercise P8 Goal: Learn how to recognize CPU spikes, invalid pointers
and disassemble code
Patterns: Exception Thread; Wild Code; CPU Spike; Multiple Exceptions; NULL Code Pointer; Invalid Pointer; Truncated Stack Trace; Stored Exception
\AWMDA-Dumps\Exercise-P8-Analysis-process-dump-ApplicationM-64.pdf
\AWMDA-Dumps\Exercise-Legacy.P8-Analysis-process-
dump-ApplicationM-32.pdf
© 2016 Software Diagnostics Services
-
Exercise P9 Goal: Learn how to recognize critical section waits and
deadlocks, dump raw stack data and see hidden exceptions
Patterns: Wait Chain; Deadlock; Hidden Exception
\AWMDA-Dumps\Exercise-P9-Analysis-process-dump-ApplicationN-64.pdf
\AWMDA-Dumps\Exercise-Legacy.P9-Analysis-process-
dump-ApplicationN-64.pdf
© 2016 Software Diagnostics Services
-
Deadlock
© 2016 Software Diagnostics Services
Critical Section00007ff676f399b0
Critical Section00007ff676f399d8
Thread 2
Thread 2 (owns)
Thread 1
Thread 1(owns)
Thread 2 (waiting)
Thread 1(waiting)
-
Exercise P10 Goal: Learn how to recognize application heap problems,
buffer and stack overflow patterns and analyze raw stack data
Patterns: Double Free; Local Buffer Overflow; Stack Overflow
\AWMDA-Dumps\Exercise-P10-Analysis-process-dump-ApplicationO-64.pdf
\AWMDA-Dumps\Exercise-Legacy.P10-Analysis-process-
dump-ApplicationO-64.pdf
© 2016 Software Diagnostics Services
-
Exercise P11 Goal: Learn how to analyze various patterns, raw stacks and
execution residue
Patterns: Divide by Zero; C++ Exception; Multiple Exceptions; Execution Residue
\AWMDA-Dumps\Exercise-P11-Analysis-process-dump-ApplicationP-64.pdf
\AWMDA-Dumps\Exercise-Legacy.P11-Analysis-process-
dump-ApplicationP-32.pdf
© 2016 Software Diagnostics Services
-
Exercise P12 Goal: Learn how to load the correct .NET WinDbg extension
and analyze managed space
Patterns: CLR Thread; Version-Specific Extension; Managed Code Exception; Managed Stack Trace
\AWMDA-Dumps\Exercise-P12-Analysis-process-dump-ApplicationR-32.pdf
© 2016 Software Diagnostics Services
-
Exercise P13 Goal: Learn how to analyze 32-process saved as a 64-bit
process memory dump
Patterns: Virtualized Process; Message Box; Execution Residue
\AWMDA-Dumps\Exercise-P13-Analysis-process-dump-ApplicationA-64.pdf
\AWMDA-Dumps\Exercise-Legacy.P13-Analysis-process-
dump-ApplicationA-32.pdf
© 2016 Software Diagnostics Services
-
Exercise P14 Goal: Learn how to analyze process memory leaks
Patterns: Spiking Thread; Thread Age; Memory Leak
(process heap)
\AWMDA-Dumps\Exercise-P14-Analysis-process-dump-ApplicationS-64.pdf
\AWMDA-Dumps\Exercise-Legacy.P14-Analysis-process-
dump-ApplicationS-32.pdf
© 2016 Software Diagnostics Services
-
Parameters and Locals Debugging TV Frames episode 0x18
© 2016 Software Diagnostics Services
http://www.debugging.tv/
-
Symbol Types Exported and imported names
Function and variable names
Data types
© 2016 Software Diagnostics Services
EXE DLL
-
Exercise P15 Goal: Learn how to navigate function parameters in cases of
reduced symbolic information in 32-bit process memory dumps
Patterns: Reduced Symbolic Information
\AWMDA-Dumps\Exercise-P15-Analysis-process-dump-notepad-32.pdf
\AWMDA-Dumps\Exercise-Legacy.P15-Analysis-process-
dump-notepad-32.pdf
© 2016 Software Diagnostics Services
-
Exercise P16 Goal: Learn how to navigate function parameters in x64
process memory dumps
Patterns: False Function Parameters, Injected Symbols
\AWMDA-Dumps\Exercise-P16-Analysis-process-dump-notepad-64.pdf
\AWMDA-Dumps\Exercise-Legacy.P16-Analysis-process-
dump-notepad-64.pdf
© 2016 Software Diagnostics Services
-
Exercise P17 Goal: Learn how to navigate object wait chains in 32-bit
memory dumps saved with ProcDump
Patterns: Wait Chain, Execution Residue, Deadlock
\AWMDA-Dumps\Exercise-P17-Analysis-process-dump-ApplicationQ-32.pdf
\AWMDA-Dumps\Exercise-Legacy.P17-Analysis-process-
dump-ApplicationQ-32.pdf
© 2016 Software Diagnostics Services
-
Pattern Links Spiking Thread CLR Thread C++ Exception Critical Section Deadlock Divide by Zero Double Free Heap Corruption Exception Stack Trace Execution Residue Hidden Exception Invalid Pointer Local Buffer Overflow Manual Dump Managed Code Exception Managed Stack Trace Multiple Exceptions Not My Version NULL Data Pointer NULL Code Pointer Stack Trace Stack Trace Collection Stack Overflow Environment Hint Wild Code Unknown Component Wait Chain Virtualized Process Message Box Version-Specific Extension Memory Leak False Function Parameters Injected Symbols Reduced Symbolic Information Truncated Stack Trace Stored Exception
© 2016 Software Diagnostics Services
http://www.dumpanalysis.org/blog/index.php/2007/05/11/crash-dump-analysis-patterns-part-14/http://www.dumpanalysis.org/blog/index.php/2009/12/07/crash-dump-analysis-patterns-part-95/http://www.dumpanalysis.org/blog/index.php/2008/10/21/crash-dump-analysis-patterns-part-77/http://www.dumpanalysis.org/blog/index.php/2007/02/09/crash-dump-analysis-patterns-part-9a/http://www.dumpanalysis.org/blog/index.php/2008/12/01/crash-dump-analysis-patterns-part-78a/http://www.dumpanalysis.org/blog/index.php/2007/08/19/crash-dump-analysis-patterns-part-23a/http://www.dumpanalysis.org/blog/index.php/2006/10/31/crash-dump-analysis-patterns-part-2/http://www.dumpanalysis.org/blog/index.php/2008/04/29/crash-dump-analysis-patterns-part-60/http://www.dumpanalysis.org/blog/index.php/2007/02/02/crash-dump-analysis-patterns-part-8/http://www.dumpanalysis.org/blog/index.php/2006/12/18/crash-dump-analysis-patterns-part-6/http://www.dumpanalysis.org/blog/index.php/2007/11/14/crash-dump-analysis-patterns-part-36/http://www.dumpanalysis.org/blog/index.php/2007/12/17/crash-dump-analysis-patterns-part-41b/http://www.dumpanalysis.org/blog/index.php/2007/07/20/crash-dump-analysis-patterns-part-17/http://www.dumpanalysis.org/blog/index.php/2011/06/17/crash-dump-analysis-patterns-part-139/http://www.dumpanalysis.org/blog/index.php/2006/10/30/crash-dump-analysis-patterns-part-1/http://www.dumpanalysis.org/blog/index.php/2008/06/19/crash-dump-analysis-patterns-part-65/http://www.dumpanalysis.org/blog/index.php/2009/04/14/crash-dump-analysis-patterns-part-6b/http://www.dumpanalysis.org/blog/index.php/2008/04/28/crash-dump-analysis-patterns-part-6a/http://www.dumpanalysis.org/blog/index.php/2007/09/10/crash-dump-analysis-patterns-part-25/http://www.dumpanalysis.org/blog/index.php/2007/09/14/crash-dump-analysis-patterns-part-27/http://www.dumpanalysis.org/blog/index.php/2008/06/10/crash-dump-analysis-patterns-part-16b/http://www.dumpanalysis.org/blog/index.php/2010/12/24/crash-dump-analysis-patterns-part-124/http://www.dumpanalysis.org/blog/index.php/2008/03/27/crash-dump-analysis-patterns-part-56/http://www.dumpanalysis.org/blog/index.php/2007/08/16/crash-dump-analysis-patterns-part-22/http://www.dumpanalysis.org/blog/index.php/2007/12/19/crash-dump-analysis-patterns-part-42b/http://www.dumpanalysis.org/blog/index.php/2007/09/11/crash-dump-analysis-patterns-part-26/http://www.dumpanalysis.org/blog/index.php/2008/02/19/crash-dump-analysis-patterns-part-51/http://www.dumpanalysis.org/blog/index.php/2011/06/01/crash-dump-analysis-patterns-part-136/http://www.dumpanalysis.org/blog/index.php/2007/08/06/crash-dump-analysis-patterns-part-20a/http://www.dumpanalysis.org/blog/index.php/2008/02/15/crash-dump-analysis-patterns-part-50/http://www.dumpanalysis.org/blog/index.php/2013/02/27/crash-dump-analysis-patterns-part-197/http://www.dumpanalysis.org/blog/index.php/2013/02/26/crash-dump-analysis-patterns-part-196/http://www.dumpanalysis.org/blog/index.php/2011/03/20/crash-dump-analysis-patterns-part-133http://www.dumpanalysis.org/blog/index.php/2012/05/23/crash-dump-analysis-patterns-part-175/
-
Kernel Memory Dumps
Exercises K1 – K5
© 2016 Software Diagnostics Services
-
Exercise K1 Goal: Learn how to get various information related to
hardware, system, sessions, processes, threads and modules
Patterns: NULL Pointer; False Effective Address; Invalid Pointer; Virtualized System; Stack Trace Collection
\AWMDA-Dumps\Exercise-K1-Analysis-normal-kernel-dump-64.pdf
\AWMDA-Dumps\Exercise-Legacy.K1-Analysis-normal-
kernel-dump-32.pdf
© 2016 Software Diagnostics Services
-
Exercise K2 Goal: Learn how to check and compare kernel pool usage
Patterns: Manual Dump; Insufficient Memory (kernel pool)
\AWMDA-Dumps\Exercise-K2-Analysis-kernel-dump-leak-
64.pdf \AWMDA-Dumps\Exercise-Legacy.K2-Analysis-kernel-dump-
leak-32.pdf
© 2016 Software Diagnostics Services
-
Exercise K3 Goal: Learn how to recognize pool corruption and check
pool data
Patterns: Dynamic Memory Corruption (kernel pool); Regular Data; Execution Residue
\AWMDA-Dumps\Exercise-K3-Analysis-kernel-dump-pool-corruption-64.pdf
\AWMDA-Dumps\Exercise-Legacy.K3-Analysis-kernel-dump-
pool-corruption-32.pdf
© 2016 Software Diagnostics Services
-
Exercise K4 Goal: Learn how to check memory access violations,
hooked or invalid code, and kernel raw stack
Patterns: Invalid Pointer; Hooked Functions (kernel space); Execution Residue; Coincidental Symbolic Information; Past Stack Trace; Rough Stack Trace; Effect Component
\AWMDA-Dumps\Exercise-K4-Analysis-kernel-dump-code-corruption-64.pdf
\AWMDA-Dumps\Exercise-Legacy.K4-Analysis-kernel-dump-
code-corruption-32.pdf
© 2016 Software Diagnostics Services
-
Exercise K5 Goal: Learn how to check I/O requests
Patterns: Blocking File; One-Thread Process
\AWMDA-Dumps\Exercise-K5-Analysis-kernel-dump-hang-
io-64.pdf \AWMDA-Dumps\Exercise-Legacy.K5-Analysis-kernel-dump-
hang-io-32.pdf
© 2016 Software Diagnostics Services
-
Pattern Links Manual Dump Invalid Pointer Virtualized System Stack Trace Collection Insufficient Memory Dynamic Memory Corruption Execution Residue Null Pointer Hooked Functions Coincidental Symbolic Information Blocking File Regular Data Past Stack Trace Rough Stack Trace Effect Component False Effective Address One-Thread Process
© 2016 Software Diagnostics Services
http://www.dumpanalysis.org/blog/index.php/2007/12/12/crash-dump-analysis-patterns-part-41a/http://www.dumpanalysis.org/blog/index.php/2006/12/18/crash-dump-analysis-patterns-part-6/http://www.dumpanalysis.org/blog/index.php/2009/07/10/crash-dump-analysis-patterns-part-87/http://www.dumpanalysis.org/blog/index.php/2007/09/14/crash-dump-analysis-patterns-part-27/http://www.dumpanalysis.org/blog/index.php/2007/11/02/crash-dump-analysis-patterns-part-13c/http://www.dumpanalysis.org/blog/index.php/2008/03/13/crash-dump-analysis-patterns-part-2b/http://www.dumpanalysis.org/blog/index.php/2008/04/29/crash-dump-analysis-patterns-part-60/http://www.dumpanalysis.org/blog/index.php/2009/04/14/crash-dump-analysis-patterns-part-6b/http://www.dumpanalysis.org/blog/index.php/2010/05/07/crash-dump-analysis-patterns-part-38b/http://www.dumpanalysis.org/blog/index.php/2007/08/30/crash-dump-analysis-patterns-part-24/http://www.dumpanalysis.org/blog/index.php/2011/06/25/crash-dump-analysis-patterns-part-145/http://www.dumpanalysis.org/blog/index.php/2012/02/12/crash-dump-analysis-patterns-part-167/http://www.dumpanalysis.org/blog/index.php/2014/10/08/crash-dump-analysis-patterns-part-214/http://www.dumpanalysis.org/blog/index.php/2014/10/07/crash-dump-analysis-patterns-part-213/http://www.dumpanalysis.org/blog/index.php/2009/10/23/crash-dump-analysis-patterns-part-88/http://www.dumpanalysis.org/blog/index.php/2014/04/26/crash-dump-analysis-patterns-part-205/http://www.dumpanalysis.org/blog/index.php/2013/05/28/crash-dump-analysis-patterns-part-199/
-
Additional Pattern Links ERESOURCE patterns and case studies Wait Chain (Executive Resources) pattern is now reprinted in this course
from Memory Dump Analysis Anthology, Volume 2, pages 147 – 150 © 2016 Software Diagnostics Services
http://www.dumpanalysis.org/blog/index.php/2011/11/07/eresource-patterns-and-case-studies/http://www.dumpanalysis.org/blog/index.php/2011/11/07/eresource-patterns-and-case-studies/http://www.dumpanalysis.org/blog/index.php/2011/11/07/eresource-patterns-and-case-studies/http://www.dumpanalysis.org/blog/index.php/2011/11/07/eresource-patterns-and-case-studies/http://www.dumpanalysis.org/blog/index.php/2011/11/07/eresource-patterns-and-case-studies/
-
Complete Memory Dumps
Exercises C1 – C4
© 2016 Software Diagnostics Services
-
Memory Spaces Complete memory == Physical memory We always see the current process space Kernel space is the same for any process
Context switch
WinDbg Commands switching to a different process context: .process /r /p
© 2016 Software Diagnostics Services
User Space
current process A (NotMyFault.exe)
Kernel Space
User Space
current process B (svchost.exe)
Kernel Space
-
Major Challenges Multiple processes (user spaces) to examine User space view needs to be correct when we examine another thread
User Space
WinDbg Commands dump all stack traces: !process 0 3f
© 2016 Software Diagnostics Services
-
Common Commands .logopen Opens a log file to save all subsequent output
View commands Dump everything or selected processes and threads (context changes automatically)
Switch commands Switch to a specific process or thread for a fine-grain analysis
© 2016 Software Diagnostics Services
-
View Commands !process 0 3f Lists all processes (including times, environment, modules) and their thread stack traces
!process 0 1f The same as the previous command but without PEB information (more secure)
!process 3f or !process 1f The same as the previous commands but only for an individual process
!thread 1f Shows thread information and stack trace
!thread 16 The same as the previous command but shows the first 3 parameters for every function
© 2016 Software Diagnostics Services
-
Switch Commands .process /r /p Switches to a specified process. Its context becomes current. Reloads symbol files for user space.
Now we can use commands like !cs 0: kd> .process /r /p fffffa80044d8b30 Implicit process is now fffffa80`044d8b30 Loading User Symbols .................................
.thread Switches to a specified thread. Assumes the current process context Now we can use commands like k*
.thread /r /p The same as the previous command but makes the thread process context current and reloads
symbol files for user space:
0: kd> .thread /r /p fffffa80051b7060 Implicit thread is now fffffa80`051b7060 Implicit process is now fffffa80`044d8b30 Loading User Symbols .................................
© 2016 Software Diagnostics Services
-
Exercise C1 Goal: Learn how to get various information related to
processes, threads and modules
Patterns: Stack Trace Collection
\AWMDA-Dumps\Exercise-C1-Analysis-normal-complete-dump-64.pdf
AWMDA-Dumps\Exercise-Legacy.C1-Analysis-normal-
complete-dump-32.pdf
© 2016 Software Diagnostics Services
-
Exercise C2 Goal: Learn how to recognize various abnormal software
behavior patterns
Patterns: Special Process; Handle Leak; Spiking Thread; Paged Out Data; Zombie Processes; Wait Chain; Dialog Box; Suspended Thread
\AWMDA-Dumps\Exercise-C2-Analysis-problem-complete-dump-64.pdf
\AWMDA-Dumps\Exercise-Legacy.C2-Analysis-problem-
complete-dump-32.pdf
© 2016 Software Diagnostics Services
-
Exercise C3 Goal: Learn how to recognize various abnormal software
behavior patterns
Patterns: Stack Trace Collection; Message Box; Wait Chain; Exception Thread
\AWMDA-Dumps\Exercise-C3-Analysis-problem-complete-dump-64.pdf
© 2016 Software Diagnostics Services
-
Wait Chain
© 2016 Software Diagnostics Services
Critical Section00007ff6590d5940
Critical Section00007ff6590d5968
Threadffffe00017a83080
Threadffffe00017a83080
(owns)
Threadffffe00017a88080
Thread ffffe00017a88080
(owns)
Threadffffe00017a83080
(waiting)
ProcessApplicationC
ProcessApplicationB
Mutantffffe00019be39f0
Threadffffe00019be4080
Threadffffe00019be4080
(owns)
Thread ffffe00017a88080
(waiting)
Threadffffe00017a79740
(waiting)
-
Exercise C4 Goal: Learn how to recognize various abnormal software
behavior patterns in x64 memory dumps
Patterns: Virtualized Process; Message Box; Frozen Process; Wait Chain (ALPC)
\AWMDA-Dumps\Exercise-C4-Analysis-problem-complete-dump-64.pdf
© 2016 Software Diagnostics Services
-
Active Memory Dump
Exercise A1
© 2016 Software Diagnostics Services
-
Exercise A1 Goal: Get familiar with active memory dumps introduced in
Windows 10
Patterns: Stack Trace Collection; Execution Residue; Rough Stack Trace; Dual Stack Trace
\AWMDA-Dumps\Exercise-A1-Analysis-problem-active-dump-64.pdf
© 2016 Software Diagnostics Services
-
Pattern Links Special Process Handle Leak Spiking Thread Stack Trace Collection Message Box Wait Chain (critical sections) Exception Stack Trace Virtualized Process Frozen Process Wait Chain (LPC/ALPC) Zombie Processes Paged Out Data Dialog Box Suspended Thread Execution Residue Rough Stack Trace Dual Stack Trace Also another pattern is present in Legacy.C2 memory dump (not shown in the
exercise transcript):
Wait Chain (window messaging)
© 2016 Software Diagnostics Services
http://www.dumpanalysis.org/blog/index.php/2008/02/12/crash-dump-analysis-patterns-part-48/http://www.dumpanalysis.org/blog/index.php/2007/07/15/crash-dump-analysis-patterns-part-13b/http://www.dumpanalysis.org/blog/index.php/2007/05/11/crash-dump-analysis-patterns-part-14/http://www.dumpanalysis.org/blog/index.php/2007/09/14/crash-dump-analysis-patterns-part-27/http://www.dumpanalysis.org/blog/index.php/2008/02/19/crash-dump-analysis-patterns-part-51/http://www.dumpanalysis.org/blog/index.php/2007/12/19/crash-dump-analysis-patterns-part-42b/http://www.dumpanalysis.org/blog/index.php/2010/08/05/crash-dump-analysis-patterns-part-105/http://www.dumpanalysis.org/blog/index.php/2007/09/11/crash-dump-analysis-patterns-part-26/http://www.dumpanalysis.org/blog/index.php/2012/10/31/crash-dump-analysis-patterns-part-184/http://www.dumpanalysis.org/blog/index.php/2008/12/17/crash-dump-analysis-patterns-part-42e/http://www.dumpanalysis.org/blog/index.php/2008/02/28/crash-dump-analysis-patterns-part-54/http://www.dumpanalysis.org/blog/index.php/2009/02/19/crash-dump-analysis-patterns-part-81/http://www.dumpanalysis.org/blog/index.php/2011/01/29/crash-dump-analysis-patterns-part-128/http://www.dumpanalysis.org/blog/index.php/2008/02/06/crash-dump-analysis-patterns-part-47/http://www.dumpanalysis.org/blog/index.php/2008/04/29/crash-dump-analysis-patterns-part-60/http://www.dumpanalysis.org/blog/index.php/2014/10/07/crash-dump-analysis-patterns-part-213/http://www.dumpanalysis.org/blog/index.php/2010/12/21/crash-dump-analysis-patterns-part-123/http://www.dumpanalysis.org/blog/index.php/2010/12/16/crash-dump-analysis-patterns-part-42h/
-
Common Mistakes Not switching to the appropriate context Not looking at full stack traces Not looking at all stack traces Not using checklists Not looking past the first found evidence Not listing both x86 and x64 stack traces
© 2016 Software Diagnostics Services
-
Kernel Minidumps Memory Dump Analysis Anthology, Volume 1 pages 43 – 67 Now reprinted in this course
© 2016 Software Diagnostics Services
-
Pattern Classification
© 2016 Software Diagnostics Services
Space/Mode Memory dump type Hooksware Wait Chain Patterns DLL Link Patterns Insufficient Memory Patterns Contention Patterns Stack Overflow Patterns Stack Trace Patterns Symbol Patterns Exception Patterns Meta-Memory Dump Patterns Module Patterns Optimization Patterns Thread Patterns Process Patterns Dynamic Memory Corruption Patterns Deadlock and Livelock Patterns .NET / CLR / Managed Space Patterns Executive Resource Patterns Falsity and Coincidence Patterns RPC, LPC and ALPC Patterns
http://www.dumpanalysis.org/blog/index.php/2008/07/21/cda-pattern-classification-spacemode/http://www.dumpanalysis.org/blog/index.php/2008/07/21/cda-pattern-classification-dump-type/http://www.dumpanalysis.org/blog/index.php/2008/08/10/hooksware/http://www.dumpanalysis.org/blog/index.php/2009/02/17/wait-chain-patterns/http://www.dumpanalysis.org/blog/index.php/2009/02/17/dll-link-patterns/http://www.dumpanalysis.org/blog/index.php/2009/02/17/insufficient-memory-patterns/http://www.dumpanalysis.org/blog/index.php/2010/09/21/contention-patterns/http://www.dumpanalysis.org/blog/index.php/2011/03/02/stack-overflow-patterns/http://www.dumpanalysis.org/blog/index.php/2011/06/18/stack-trace-patterns/http://www.dumpanalysis.org/blog/index.php/2011/10/05/symbol-patterns/http://www.dumpanalysis.org/blog/index.php/2011/11/29/exception-patterns/http://www.dumpanalysis.org/blog/index.php/2012/03/22/meta-memory-dump-patterns/http://www.dumpanalysis.org/blog/index.php/2012/07/15/module-patterns/http://www.dumpanalysis.org/blog/index.php/2012/11/16/optimization-patterns/http://www.dumpanalysis.org/blog/index.php/2013/01/05/thread-patterns/http://www.dumpanalysis.org/blog/index.php/2013/01/05/process-patterns/http://www.dumpanalysis.org/blog/index.php/2009/02/17/dynamic-memory-corruption-patterns/http://www.dumpanalysis.org/blog/index.php/2009/02/17/deadlock-patterns/http://www.dumpanalysis.org/blog/index.php/2011/04/22/net-clr-managed-space-patterns/http://www.dumpanalysis.org/blog/index.php/2011/11/07/eresource-patterns-and-case-studies/http://www.dumpanalysis.org/blog/index.php/2014/04/28/falsity-and-coincidence-patterns/http://www.dumpanalysis.org/blog/index.php/2011/11/14/rpc-lpc-and-alpc-patterns-and-case-studies/
-
Pattern Case Studies 70 multiple pattern case studies: http://www.dumpanalysis.org/blog/index.php/pattern-cooperation/
Pattern Interaction chapters in Memory Dump Analysis Anthology
© 2016 Software Diagnostics Services
http://www.dumpanalysis.org/blog/index.php/pattern-cooperation/
-
Additional Resources WinDbg Help / WinDbg.org (quick links) DumpAnalysis.org / PatternDiagnostics.org Debugging.TV / YouTube.com/DebuggingTV Windows Internals, 6th ed. Practical Foundations of Windows Debugging, Disassembling, Reversing Advanced Windows Debugging Inside Windows Debugging Windows Debugging Notebook: Essential User Space WinDbg Commands Memory Dump Analysis Anthology
© 2016 Software Diagnostics Services
http://www.dumpanalysis.org/practical-foundations-windows-debugging-disassembling-reversinghttp://www.dumpanalysis.org/advanced-software-debugging-reference
-
Further Training Courses Practical Foundations of Windows Debugging, Disassembling, Reversing
Advanced Windows Memory Dump Analysis with Data Structures, 2nd edition
Accelerated .NET Memory Dump Analysis, 2nd edition
Accelerated Windows Malware Analysis with Memory Dumps
Accelerated Disassembly, Reconstruction and Reversing
Accelerated Windows Debugging3
© 2016 Software Diagnostics Services
http://www.patterndiagnostics.com/practical-foundations-windows-debugging-disassembling-reversinghttp://www.patterndiagnostics.com/advanced-windows-memory-dump-analysis-bookhttp://www.patterndiagnostics.com/advanced-windows-memory-dump-analysis-bookhttp://www.patterndiagnostics.com/advanced-windows-memory-dump-analysis-bookhttp://www.patterndiagnostics.com/accelerated-net-memory-dump-analysis-bookhttp://www.patterndiagnostics.com/accelerated-net-memory-dump-analysis-bookhttp://www.patterndiagnostics.com/accelerated-net-memory-dump-analysis-bookhttp://www.patterndiagnostics.com/accelerated-malware-analysishttp://www.patterndiagnostics.com/accelerated-malware-analysishttp://www.patterndiagnostics.com/accelerated-disassembly-reconstruction-reversinghttp://www.patterndiagnostics.com/accelerated-disassembly-reconstruction-reversinghttp://www.patterndiagnostics.com/accelerated-windows-debugging-3http://www.patterndiagnostics.com/accelerated-windows-debugging-3http://www.patterndiagnostics.com/accelerated-windows-debugging-3
-
Q&A
Please send your feedback using the contact form on PatternDiagnostics.com
© 2016 Software Diagnostics Services
-
Thank you for attendance!
© 2016 Software Diagnostics Services
Accelerated �PrerequisitesTraining GoalsTraining PrinciplesCoverageMain Schedule SummaryOptional Schedule SummaryPart 1: FundamentalsProcess Space (x86)Process Space (x64)Application/Process/ModuleOS Kernel/Driver/ModuleProcess Virtual SpaceProcess Memory DumpKernel Memory DumpComplete Memory DumpProcess ThreadsSystem ThreadsThread Stack Raw DataThread Stack TraceThread Stack Trace (no PDB)Exceptions (Access Violation)Exceptions (Runtime)Pattern-Oriented Diagnostic AnalysisPart 2: Practice ExercisesLinksExercise 0Process Memory DumpsExercise P1Exercise P2Exercise P3Exercise P4Exercise P5Exercise P6Exercise P7Exercise P8Exercise P9DeadlockExercise P10Exercise P11Exercise P12Exercise P13Exercise P14Parameters and LocalsSymbol TypesExercise P15Exercise P16Exercise P17Pattern LinksKernel Memory DumpsExercise K1Exercise K2Exercise K3Exercise K4Exercise K5Pattern LinksAdditional Pattern LinksComplete Memory DumpsMemory SpacesMajor ChallengesCommon CommandsView CommandsSwitch CommandsExercise C1Exercise C2Exercise C3Wait ChainExercise C4Active Memory DumpExercise A1Pattern LinksCommon MistakesKernel MinidumpsPattern ClassificationPattern Case StudiesAdditional ResourcesFurther Training CoursesQ&ASlide Number 79