Bill Au
CBS Interactive
http://www.apache.org/~billa/oscon2012/
Troubleshooting Slow or Hung Java Applications
2
Agenda
» Tools of the Trade
» Case Studies
2
3
Tools – Out Of The Box
» jinfo, jmap, jstack, jstat
» JConsole, VisualVM – Monitor CPU usage by using JTop plug-in ($JAVA_HOME/
demo/management/JTop)
» java.lang.management – Platform MXBeans: runtime, operating system, class loading,
compilation, garbage collection, memory, thread – Sample code in $JAVA_HOME/demo/management – JDK demo and sample was part of JDK download for
jdk6u31 or older, separate download for u32+
3
4
Tools – java.lang.management
» Sample code: import java.lang.management.*;
ThreadMXBean tb=ManagementFactory.getThreadMXBean();
if (tb.isThreadCpuTimeSupported()) {
long[] tids=tb.getAllThreadIds();
for (long tid : tids) {
System.out.println(“cputime:”+tb.getThreadCpuTime(tid));
System.out.println(“usertime:”+tb.getThreadUserTime(tid));
}
}
4
5
Tools
» Garbage collection log – -Xloggc:<filename> – Timestamp and duration of GC events – Sizes before and after GC events – More options to explore: − -XX:+PrintGCDateStamps − -XX:+PrintGCDetails − -XX:+PrintTenuringDistribution − -XX:+PrintGCApplicationsStoppedTime
– Some GC flags can be set dynamically by jinfo/JConsole/VisualVM: PrintGC, PrintGCDetails, PrintGCTimeStamps
» visualgc – GUI for garbage collection monitoring
5
6
Tools
» Thread dump – kill -3 (or kill –SIGHUP) – JConsole, VisualVM – ThreadMXBean – jstack (-F)
» Consecutive thread dumps – Stop motion animation
6
7
Hung or Slow App
» Long GC pauses and/or high GC overhead – Basic GC tuning − max heap size: -Xmx − type of collector used
− -XX:+UserSerialGC − -XX:+UseParallelGC/-XX:+UseParallelOldGC − -XX:+UseConcMarkSweepGC − -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC (jdk6u14+)
– Advanced GC tuning − young generation size: -Xmn
– Baseline, tune, measure one change at a time
» Demo: HPjmeter – garbage collection analysis (and more…)
7
8
Hung or Slow App
» Deadlock – Take thread dump to run the JVM’s deadlock detector
» Looping Threads – Monitor CPU times of running threads − ThreadMXBean − JTop plug-in for JConsole/VisualVM
– Inspect stack trace of suspicious threads
8
9
Slow or Hung App
» Blocked threads – Look for blocked threads waiting on the same condition – Identify bottleneck for resource contention – Inspect code for incautious and/or improper synchronization
» Demo: samurai – thread dump analysis tool
» Stuck threads – Look for runnable threads with the same stack trace across
consecutive thread dumps – Some common causes: − Blocking for a network call without specifying connect and/or
read timeout (default may be no timeout) − Long execution time for calls to backend servers
9
10
Links
» Presentation http://www.apache.org/~billa/oscon2012
» HPjmeter https://h20392.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=HPJMETER
» samurai http://yusuke.homeip.net/samurai/en/index.html https://github.com/yusuke/samurai
10
Top Related