Memory and runtime analysis
-
Upload
admexoplatform -
Category
Technology
-
view
117 -
download
5
description
Transcript of Memory and runtime analysis
Click to edit the outline text format
Second Outline Level
Third Outline Level Fourth Outline Level
Fifth Outline Level Sixth Outline Level Seventh Outline Level Eighth Outline Level
Ninth Outline LevelClick to edit subtitle
Click to edit the outline text format
Second Outline Level
Third Outline Level
Fourth Outline Level Fifth Outline Level Sixth Outline Level Seventh Outline Level Eighth Outline Level
Ninth Outline LevelClick to edit date
Click to edit the outline text format
Second Outline Level
Third Outline Level Fourth Outline Level
Fifth Outline Level Sixth Outline Level Seventh Outline Level Eighth Outline Level
Ninth Outline LevelClick to edit presenter name, title
Java PerformanceMemory and Runtime Analysis
March 2012
Viet Nguyen, eXo CT
www.exoplatform.com - Copyright 2012 eXo Platform 2
1
2
3
4
Click to edit the outline text format
Second Outline Level
Third Outline Level Fourth Outline Level
Fifth Outline Level Sixth Outline Level Seventh Outline Level Eighth Outline Level
− Ninth Outline LevelAgenda item 1− Second level
Click to edit the outline text format
Second Outline Level
Third Outline Level Fourth Outline Level
Fifth Outline Level Sixth Outline Level Seventh Outline Level Eighth Outline Level
− Ninth Outline LevelAgenda item 2− Second level
Click to edit the outline text format
Second Outline Level
Third Outline Level Fourth Outline Level
Fifth Outline Level Sixth Outline Level Seventh Outline Level Eighth Outline Level
− Ninth Outline LevelAgenda item 3− Second level
Agenda
− Java runtime profiling− JvisualVM & JProfiler− eXo JCR statistic tool
− Introduction− Java Memory Model− Heap dump
− Java Memory Problems− Common problems− Memory Leak
− Analysing Heap Dumps with Eclipse Memory Analyzer (MAT)− Shallow & Retained Size, Dominator Tree− Analysing Heap dumps with MAT
www.exoplatform.com - Copyright 2012 eXo Platform
Click to edit the outline text format
Second Outline Level
Third Outline Level Fourth Outline Level
Fifth Outline Level
Sixth Outline Level
Seventh Outline Level
Eighth Outline Level
− Ninth Outline LevelClick to edit first level• Second level• Third level
• Fourth level
• Fifth level
3
Code Performance Questions
− Did you choose the right data structure ?− Are you using the right sorting algorithm ?− Is your recursive method TOO recursive ?− Are you throwing away a computation that could prove useful
later ?− Are you creating too many objects unnecessarily or otherwise
wasting memory ?
www.exoplatform.com - Copyright 2012 eXo Platform
Click to edit the outline text format
Second Outline Level
Third Outline Level Fourth Outline Level
Fifth Outline Level
Sixth Outline Level
Seventh Outline Level
Eighth Outline Level
− Ninth Outline LevelClick to edit first level• Second level• Third level
• Fourth level
• Fifth level
4
Optimization Metrics
− Runtime / CPU usage− What line of code the program is spending the most time in− What call / invocation paths were used to get to these lines
− Memory usage− What kind of objects are sitting on the heap− Where were they allocated− Who is pointing to them now− “Memory leak”
www.exoplatform.com - Copyright 2012 eXo Platform
Click to edit the outline text format
Second Outline Level
Third Outline Level Fourth Outline Level
Fifth Outline Level
Sixth Outline Level
Seventh Outline Level
Eighth Outline Level
− Ninth Outline LevelClick to edit first level• Second level• Third level
• Fourth level
• Fifth level
5
Java's Memory Model
www.exoplatform.com - Copyright 2012 eXo Platform
Click to edit the outline text format
Second Outline Level
Third Outline Level Fourth Outline Level
Fifth Outline Level
Sixth Outline Level
Seventh Outline Level
Eighth Outline Level
− Ninth Outline LevelClick to edit first level• Second level• Third level
• Fourth level
• Fifth level
6
Java's Memory Model 2
www.exoplatform.com - Copyright 2012 eXo Platform
Click to edit the outline text format
Second Outline Level
Third Outline Level Fourth Outline Level
Fifth Outline Level
Sixth Outline Level
Seventh Outline Level
Eighth Outline Level
− Ninth Outline LevelClick to edit first level• Second level• Third level
• Fourth level
• Fifth level
7
The JVM Heaps
‘Thread Stacks
Buffers
JIT Compiled Code
Motif structures
Size
Next
Size
Next
freelist
Nullfree storage
free storage
Native Heap
Java Heap
Free List
Xms - Active Area of Heap
www.exoplatform.com - Copyright 2012 eXo Platform
Click to edit the outline text format
Second Outline Level
Third Outline Level Fourth Outline Level
Fifth Outline Level
Sixth Outline Level
Seventh Outline Level
Eighth Outline Level
− Ninth Outline LevelClick to edit first level• Second level• Third level
• Fourth level
• Fifth level
8
The “Correct” Java Heap Size
•GC will adapt heap size to keep occupancy between 40% and 70%– Heap occupancy over 70% causes frequent GC cycles
•Which generally means reduced performance– Heap occupancy below 40% means infrequent GC cycles,
but cycles longer than they needs to be•Which means longer pause times that necessary•Which generally means reduced performance
•The maximum heap size setting should therefore be 43% larger than the maximum occupancy of the application
–Eg. For 70MB occupancy, 100MB Max heap is required (which is 70MB + 43% of 70MB)
www.exoplatform.com - Copyright 2012 eXo Platform 9
Binary Heap Dump
A heap dump is a snapshot of objects that are alive at one point in time.
It contains: Objects: fields, references, primitive values, … Classes: class loader, super class, static fields, …
A heap dump does not contain where the objects have been created which objects have been garbage collected
IOIOIIIOIIOIOIOIOIIOIIOI
IOIOIIIOIIOIOIOIOIIOIIOI
www.exoplatform.com - Copyright 2012 eXo Platform
Click to edit the outline text format
Second Outline Level
Third Outline Level Fourth Outline Level
Fifth Outline Level
Sixth Outline Level
Seventh Outline Level
Eighth Outline Level
− Ninth Outline LevelClick to edit first level• Second level• Third level
• Fourth level
• Fifth level
10
Heap Dumps Are Useful for Analysis
− Heap dumps come “for free”− They are suitable for production− The comprehensive data allows detail analysis− There is a wide platform coverage
− HPROF dumps from HotSpot based JVMs
− DTFJ and PHD dumps from IBM JVMs
www.exoplatform.com - Copyright 2012 eXo Platform 11
How To Get A Heap Dump
Non-Interactive
-XX:+HeapDumpOnOutOfMemoryError
On Demand
JDK1.4.2_12 and -XX:+HeapDumpOnCtrlBreak
JDK6 and Jconsole
More…http://wiki.eclipse.org/index.php/MemoryAnalyzer#Getting_a_Heap_Dump
Click to edit the outline text format
Second Outline Level
Third Outline Level Fourth Outline Level
Fifth Outline Level Sixth Outline Level Seventh Outline Level Eighth Outline Level
Ninth Outline LevelClick to edit subtitle
Java Memory ProblemsCommon problemsMemory Leak
www.exoplatform.com - Copyright 2012 eXo Platform
Click to edit the outline text format
Second Outline Level
Third Outline Level Fourth Outline Level
Fifth Outline Level
Sixth Outline Level
Seventh Outline Level
Eighth Outline Level
− Ninth Outline LevelClick to edit first level• Second level• Third level
• Fourth level
• Fifth level
13
Java Memory Problems
− Memory leaks• Referencing objects that are no
longer used• Developer forget clear them
− Inefficient object creation• GC must constantly clean up the
heap• High CPU consumption by GC• Application response times
increases
− Unnecessarily high memory usage• Implementations consuming too
much memory• Large amount of state information• Unbound or inefficiently configured
caches• Large temporary objects: big
documents (XML, PDF, images...) are read and processed
− Inefficient garbage collector behaviour• Missing or wrong configuration of
GC
www.exoplatform.com - Copyright 2012 eXo Platform
Click to edit the outline text format
Second Outline Level
Third Outline Level Fourth Outline Level
Fifth Outline Level
Sixth Outline Level
Seventh Outline Level
Eighth Outline Level
− Ninth Outline LevelClick to edit first level• Second level• Third level
• Fourth level
• Fifth level
14
Most Common Memory Leaks
− Thread Local Variables• ThreadLocals are often used to track
state• Threads are often never garbage
collected• Memory leaks may happen if thread
local variable is not clean carefully
− Circular and complex bi-directional references• All objects will not be garbage
collected
− Mutable static fields and collections• Most common reason• Statics often used to hold caches or
share states• Memory leak happens easily if this is
not done diligently
− Wrong implementation of equals/hashcode• Lead to memory leaks when used as
a key in a map
www.exoplatform.com - Copyright 2012 eXo Platform
Click to edit the outline text format
Second Outline Level
Third Outline Level Fourth Outline Level
Fifth Outline Level
Sixth Outline Level
Seventh Outline Level
Eighth Outline Level
− Ninth Outline LevelClick to edit first level• Second level• Third level
• Fourth level
• Fifth level
15
Most Common Memory Leaks
− Classloader Leaks• Classes are referenced by their
class loader• They will not get garbage collected
until classloader itself is collected• First leak form: an object whose
class belongs to the class loader is still referenced by a cache, a thread local …
• Second leak form: Some frameworks allow dynamic creation of new classes. If they are forgotten, PermGen leaks happen
− JNI memory leaks• JNI is used to call native code• Native code can handle, call, or
create java objects• These objects are referenced until
the native method returns• First leak form: native method run
forever• Second leak form: You keep created
objects after native code has ended
www.exoplatform.com - Copyright 2012 eXo Platform
Click to edit the outline text format
Second Outline Level
Third Outline Level Fourth Outline Level
Fifth Outline Level
Sixth Outline Level
Seventh Outline Level
Eighth Outline Level
− Ninth Outline LevelClick to edit first level• Second level• Third level
• Fourth level
• Fifth level
16
How to interpret the Memory Leak Analysis Results
What is leaking?
– What is the object (e.g. a HashMap) holding all the leaking objects i.e. leak container.
– What are the objects getting added to the leak container i.e. leak unit.
– Who is holding the leak container in memory? What are the object types and package names of objects on the chain of references from a root object to the leak container i.e. owner chain.
www.exoplatform.com - Copyright 2012 eXo Platform
Click to edit the outline text format
Second Outline Level
Third Outline Level Fourth Outline Level
Fifth Outline Level
Sixth Outline Level
Seventh Outline Level
Eighth Outline Level
− Ninth Outline LevelClick to edit first level• Second level• Third level
• Fourth level
• Fifth level
17
View of the leaking data structure
Significant entities
– An owner chain
– A leak root
– A container
– The unit of the leak
– Leak contents
MyClass
HashSet
HashMap
HM$Entry
String
Char[]
HM$Entry
String
Char[]
HM$Entry
String
Char[]
Owner Chain
Leak Root
Container
Contents
Leaking Unit
www.exoplatform.com - Copyright 2012 eXo Platform
Click to edit the outline text format
Second Outline Level
Third Outline Level Fourth Outline Level
Fifth Outline Level
Sixth Outline Level
Seventh Outline Level
Eighth Outline Level
− Ninth Outline LevelClick to edit first level• Second level• Third level
• Fourth level
• Fifth level
18
Memory Leak Example
public class MyClass {
static HashSet myContainer = new HashSet();
public void leak(int numObjects) {
for (int i = 0; i < numObjects; ++i) {
String leakingUnit = new String("this is leaking object: " + i);
myContainer.add(leakingUnit);
}
}public static void main(String[] args) throws Exception {
{
MyClass myObj = new MyClass();
myObj.leak(1000000); // 1 million
}
System.gc();
}
}
Leaking Units
Leak Container
Leak
Root
Click to edit the outline text format
Second Outline Level
Third Outline Level Fourth Outline Level
Fifth Outline Level Sixth Outline Level Seventh Outline Level Eighth Outline Level
Ninth Outline LevelClick to edit subtitle
Analyzing Heap dumps with MATRetained size, Dominator TreeMAT featuresDemos
www.exoplatform.com - Copyright 2012 eXo Platform
Click to edit the outline text format
Second Outline Level
Third Outline Level Fourth Outline Level
Fifth Outline Level
Sixth Outline Level
Seventh Outline Level
Eighth Outline Level
− Ninth Outline LevelClick to edit first level• Second level• Third level
• Fourth level
• Fifth level
20
Tools For Analysing Heap Dumps
www.exoplatform.com - Copyright 2012 eXo Platform 21
Eclipse Memory Analyzer Introduction
The number of memory-related problems is very high They are extremely difficult to analyze Analysis requires expertise in the analyzed code
Eclipse Memory Analyzer:
Simplifies memory analysis Extensible Free for download @ www.eclipse.org/mat
www.exoplatform.com - Copyright 2012 eXo Platform 22
A Little Bit of History
Problem #1: multi-GB heap dumps from 64-bit machines Quickly open and re-open a heap dump on 32-bit machines
Problem #2: millions of objects. Where is the „needle“? Dominator Tree: fast calculation of retained memory Meaningful names for class loaders: see components SQL-like language: create custom queries
Problem #3: complexity of analysis Reports on memory leak suspects and checks for known
antipatterns
Problem #4: HPROF heap dumps only Support for IBM system dumps
www.exoplatform.com - Copyright 2012 eXo Platform 23
Shallow And Retained Size
Shallow heap is the memory consumed by one object Retained set of X is the set of objects that will be garbage collected
if X is garbage collected Retained heap of X is the sum of shallow sizes of all objects in the
retained set of X, i.e. memory kept alive by X
Set of elements Retained Set
C C, F, G, J
K K
C, K C, F, G, J, K, I
www.exoplatform.com - Copyright 2012 eXo Platform 24
Dominator Tree
The Dominator Tree is a Transformation of the Cyclic Object Graph into a “Keep-Alive” Tree:
Every node in the Tree is directly responsible for keeping alive its children
Object X dominates object Y if all paths from the roots to Y run through X
Benefits: Fast calculation of the retained size (sum all children) List of distinct big objects (first level of the tree) Fast identification of responsible objects (just go up the tree)
www.exoplatform.com - Copyright 2012 eXo Platform 25
MAT Features
Memory Leak Hunter
Automatically detect memory leak suspects Discover if the issue is known (and a fix available) Collect details for in depth analysis by the code experts
www.exoplatform.com - Copyright 2012 eXo Platform 26
MAT Report Overview
One big object: memory leak suspect
Any up-to-date architecture loads components with separate class loaders, be it OSGi or JEE application servers.Extensible to display meaningful names.
Search by keywords: identify if problem is known
Classification for trouble ticket system: less ping-pong of trouble tickets.
www.exoplatform.com - Copyright 2012 eXo Platform 27
MAT Report Details
Accumulated objects
A significant drop in the retained sizes shows the
accumulation point
The chain of objects and references which keep
the suspect alive
www.exoplatform.com - Copyright 2012 eXo Platform 28
MAT Features
Top Components Report
Analyze components, that occupy more memory, than a certain threshold
Give hints where memory footprint can be optimized Check for known antipatterns
www.exoplatform.com - Copyright 2012 eXo Platform 29
MAT Report Details
www.exoplatform.com - Copyright 2012 eXo Platform 30
MAT Report Details
www.exoplatform.com - Copyright 2012 eXo Platform
Click to edit the outline text format
Second Outline Level
Third Outline Level Fourth Outline Level
Fifth Outline Level
Sixth Outline Level
Seventh Outline Level
Eighth Outline Level
− Ninth Outline LevelClick to edit first level• Second level• Third level
• Fourth level
• Fifth level
31
How to Analyze Memory Leaks
− Find the biggest objects− Analyze why they are kept in memory− Analyze what make them big
Use MAT To Analyze Leaks
− Get a “good” heapdump− Find the biggest objects (in the dominator tree)− Analyze who kept them alive (using paths)− Analyze what made them big (looking at their retain set)− Use “Leak report” to automate the analysis
www.exoplatform.com - Copyright 2012 eXo Platform
Click to edit the outline text format
Second Outline Level
Third Outline Level Fourth Outline Level
Fifth Outline Level
Sixth Outline Level
Seventh Outline Level
Eighth Outline Level
− Ninth Outline LevelClick to edit first level• Second level• Third level
• Fourth level
• Fifth level
32
How to Analyze a Heavy Request
− Look at the thread attributes – name, class, etc...− Look at the local variables− Look at the stack of the thread
Use MAT To Analyze Thread Activity
− Inspect threads attributes and local variables (in object explorer)
− Analyze stack trace
www.exoplatform.com - Copyright 2012 eXo Platform
Click to edit the outline text format
Second Outline Level
Third Outline Level Fourth Outline Level
Fifth Outline Level
Sixth Outline Level
Seventh Outline Level
Eighth Outline Level
− Ninth Outline LevelClick to edit first level• Second level• Third level
• Fourth level
• Fifth level
33
Find Where Memory Footprint Can Be Optimized
− Inspect the set of retained objects− Search for inefficiently used data structures− Look for redundant data
Use MAT To Analyze Memory Footprint
− Analyze retained objects (in dominator tree, retained set)− Use “Group by Value” to find redundant objects− Use the commands from the “Collections” group− Use “Component Report” to automate the analysis
Click to edit the outline text format
Second Outline Level
Third Outline Level Fourth Outline Level
Fifth Outline Level Sixth Outline Level Seventh Outline Level Eighth Outline Level
Ninth Outline LevelClick to edit subtitle
Java Runtime ProfilingJVisualVMJProfilereXo JCR statistic tool
www.exoplatform.com - Copyright 2012 eXo Platform
Click to edit the outline text format
Second Outline Level
Third Outline Level Fourth Outline Level
Fifth Outline Level
Sixth Outline Level
Seventh Outline Level
Eighth Outline Level
− Ninth Outline LevelClick to edit first level• Second level• Third level
• Fourth level
• Fifth level
35
What to do with profiler results
− Observe which methods are being called the most− These may not necessarily be the “slowest” methods!
− Observe which methods are taking the most time relative to others− Common problem: inefficient unbuffered I/O− Common problem: poor choice of data structure− Common problem: recursion call overhead− Common problem: unnecessary re-computation of expensive
information, or unnecessary multiple I/O of same data
Click to edit the outline text format
Second Outline Level
Third Outline Level Fourth Outline Level
Fifth Outline Level Sixth Outline Level Seventh Outline Level Eighth Outline Level
Ninth Outline LevelClick to edit subtitle
Q&A
http://kohlerm.blogspot.com/search/label/memoryhttp://visualvm.java.net/docindex.html
http://docs.exoplatform.org/PLF35/index.jsp?topic=/org.exoplatform.doc.35/bk05pt01ch38s02.html
www.exoplatform.com - Copyright 2012 eXo Platform
Click to edit the outline text format
Second Outline Level
Third Outline Level Fourth Outline Level
Fifth Outline Level
Sixth Outline Level
Seventh Outline Level
Eighth Outline Level
− Ninth Outline LevelClick to edit first level• Second level• Third level
• Fourth level
• Fifth level
37
How will this session do
− Present common memory & thread-related issues− Give hints how to solve them using heap dumps and Eclipse
Memory Analyzer− Show demos using real-life heap dumps − Runtime Java profiling with JvisualVM, Jprofiler, eXo JCR
statistics tool (exo.jcr.component.statistics)