Jvm Performance Tunning
71
JavaStudy Network Daehyub Cho JVM [Java Virtual Machine] Performance Tuning
-
Upload
guest1f2740 -
Category
Technology
-
view
3.070 -
download
0
description
Java Virtual Machine Tuning guide.It describes JVM Memory model and tuning guide
Transcript of Jvm Performance Tunning
JavaStudy NetworkDaehyub Cho
JVM [Java Virtual Machine]
PerformanceTuning
AGENDA
Basic concept of JVM Tuning1
Hotspot compiler2
Threading Model3
Memory Model4
Basic Concept of JVM TuningBasic concept of JVM Tuning
Basic of performance tuning
1. Decide what performance level is “good enough”2. Test & measurement
• Scenario based• Stress Tool (Load Runner)• Profiling Tool (J probe, etc)
3. Profile application to find bottlenecks4. Tuning
• Application *• Middleware [WAS]• OS• JVM
5. Return to Step 2 [feedback]
JVM Tuning
• Improve performance about 10~20%• Find appropriate parameter for your application
– Hotspot compile option– Thread model option *– GC and memory related option **
• Changing parameter is very dangerous action– Need more test and feed back– Ref spec.org
Hotspot CompilerHotspot compiler
JVM Layout
• Hotspot from JDK 1.3
VM
ClientCompiler
ServerCompiler
• Runtime• GC• Interpreter• Threading & Locking• ….
JVM
Hotspot Compiler
Hotspot compiler
• JIT (Just-In-Time Compiler)– Compile byte code to native code– Compile as rules of optimization (Not thinking)– At execution/installation– Compile byte code to native code
• Hotspot– Compile byte code to native code– ‘Thinking’ to trying find where optimization can take place– Adaptive Optimizing in runtime
Hotspot Detection
• Hotspot detection• Method Inlining• Dynamic Deoptimization
Hotspot Detection and Method Inlining
• Literal constants are folded
• String concatenation is sometimes folded
• Constant fields are inlined
int foo = 9* 10; int foo = 90;
String foo = “Hello “ + (9*10); String foo = “Hello 90”;
public class A{ public static final VALUE=99;}public class B{ static int VALUE2=A.VALUE;}
public class B{ static int VALUE2=99;}
When after compiling class B
Hotspot detection / Method Inlining
• Dead code branches are eliminated
public class A{ static final boolean DEBUG = false; public void methodA() if(DEBUG) System.out.println(“DEBUG MODE); System.out.println(“Say Hello”); }// method A}// class A
↓public class A{ static final boolean DEBUG = false; public void methodA() System.out.println(“Say Hello”); }// method A}// class A
Hotspot Client compiler
• Java Option : -client• Focused on Simple & Fast start up• 3 Phase compiler
– HIR (High Level Intermediate Representation)– LIR (Low Level Intermediate Representation)– Machine code
• It focuses on local code quality and does very few global optimizations since those are often the most expensive in terms of compile time
• It has for inlining any function that has no exception handlers or synchronization and also supports deoptimization for debugging and inlining
Hotspot Server compiler
• Java Option : -server• Focused on optimization• SSA (Static Single Assignment)-based IR
Hotspot compiler Option
• Hotspot compile option– -XX:MaxInlineSize=<size>
• Integer specifying maximum number of bytecode instructions in a method which gets inlined.
– -XX:FreqInlineSize=<size>• Integer specifying maximum number of bytecode instructions in a
frequently executed method which gets inlined.
– -Xint• Interpreter only (no JIT compilation)
– -XX:+PrintCompilation
ThreadingThreading model
Threading Model
• Thread Model– Java is multi threaded programming language– Native thread model from JDK 1.2
• Thread mapping (M:N and 1:1)• Thread synchronization
JavaApplication
Java Thread
OperatingSystemThread Handling
Thread SchedulingLock Mgmt (synchronization)
JVM
Solaris M:N Thread Model
JavaApplication
Java Thread
JVM
Solaris OS
OS Kernel
Solaris Thread
LWP
Kernel Thread
Solaris M:N Thread Model
• Solaris M:N Thread Model– Thread based synchronization– LWP based synchronization
Thread based sync LWP based sync
JDK1.2 N/A Default
JDK1.3 Default -XX:+UseLWPSynchronization
JDK1.4 -XX:-UseLWPSynchronization Default
Solaris 1:1 Thread Model
JavaApplication
Java Thread
JVM
Solaris OS
OS Kernel
Solaris Thread
LWP
Kernel Thread
Solaris 1:1 Thread Model
• Solaris 1:1 Thread Model– Bound thread– Alternate Libthread
Bound Thread Alternate Libthread*
JDK1.2 N/A export LD_LIBRARY_PATH=/usr/lib/lwp
JDK1.3 -XX:+UseBoundThreads export LD_LIBRARY_PATH=/usr/lib/lwp
JDK1.4 -XX:+UseBoundThreads export LD_LIBRARY_PATH=/usr/lib/lwp
※ In Solaris 9, alternate lib thread is default, do not add /usr/lib/lwp to LD_LIBRARY_PATH
JVM Performance Test on Solaris
Architecture Cpus Threads Model %diff in throughput (against Standard Model)
Sparc 30 400/2000 Standard ---
Sparc 30 400/2000 LWP Synchronization 215%/800%
Sparc 30 400/2000 Bound Threads -10%/-80%
Sparc 30 400/2000 Alternate One-to-one 275%/900%
Sparc 4 400/2000 Standard ---
Sparc 4 400/2000 LWP Synchronization 30%/60%
Sparc 4 400/2000 Bound Threads -5%/-45%
Sparc 4 400/2000 Alternate One-to-one 30%/50%
Sparc 2 400/2000 Standard ---
Sparc 2 400/2000 LWP Synchronization 0%/25%
Sparc 2 400/2000 Bound Threads -30%/-40%
Sparc 2 400/2000 Alternate One-to-one -10%/0%
Intel 4 400/2000 Standard ---
Intel 4 400/2000 LWP Synchronization 25%/60%
Intel 4 400/2000 Bound Threads 0%/-10%
Intel 4 400/2000 Alternate One-to-one 20%/60%
Intel 2 400/2000 Standard ---
Intel 2 400/2000 LWP Synchronization 15%/45%
Intel 2 400/2000 Bound Threads -10%/-15%
Intel 2 400/2000 Alternate One-to-one 15%/35%
< Solaris 8 with JVM 1.3 >See next page graph!!
JVM Performance Test on Solaris
• Performance Test Result Graph
Memory TuningMemory Model
Memory Tuning
• Garbage Collection• JVM Memory Layout• Garbage Collection Model• Server VM and Client VM• Garbage Collection Measurement & Analysis• Tuning Garbage Collection
Generational Garbage Collection
JVM Memory Layout
• New/Young – Recently created object• Old – Long lived object• Perm – JVM classes and methods
Eden Old Perm
New/Young Old
Used in Application JVM
Total Heap Size
SS1 SS2
Garbage Collection
• Garbage Collection– Collecting unused java object– Cleaning memory– Minor GC
• Collection memory in New/Young generation
– Major GC (Full GC)• Collection memory in Old generation
Minor GC
• Minor Collection– New/Young Generation– Copy and Scavenge – Very Fast
Minor GC
Eden SS1 SS1
Copy live objects to Survivor area
New Object
Garbage
Lived Object
1st Minor GC
Old
Old
Old
Minor GC
2nd Minor GC
Old
Old
Old
New Object
Garbage
Lived Object
Minor GC
OLD
3rd Minor GC
Objects moved old space when they become tenured
New Object
Garbage
Lived Object
Major GC
• Major Collection– Old Generation– Mark and compact– Slow
• 1st – goes through the entire heap, marking unreachable objects• 2nd – unreachable objects are compacted
Major GC
Eden SS1 SS2
Eden SS1 SS2
Mark the objects to be removed
Eden SS1 SS2
Compact the objects to be removed
Server option versus Client option
• -X:NewRatio=2 (1.3) , -Xmn128m(1.4), -XX:NewSize=<size> -XX:MaxNewSize=<size>
GC Tuning Parameter
• Memory Tuning Parameter– Perm Size : -XX:MaxPermSize=64m– Total Heap Size : -ms512m –mx 512m– New Size
• -XX:NewRatio=2 Old/New Size• -XX:NewSize=128m• -Xmn128m (JDK 1.4)
– Survivor Size : -XX:SurvivorRatio=64 (eden/survivor)– Heap Ratio
• -XX:MaxHeapFreeRatio=70• -XX:MinHeapFreeRatio=40
– Suvivor Ratio• -XX:TargetSurvivorRatio=50
Support for –XX Option
• Options that begin with -X are nonstandard (not guaranteed to be supported on all VM implementations), and are subject to change without notice in subsequent releases of the Java 2 SDK.
• Because the -XX options have specific system requirements for correct operation and may require privileged access to system configuration parameters, they are not recommended for casual use. These options are also subject to change
without notice.
Garbage Collection Model
• New type of GC– Default Collector– Parallel GC for young generation - JDK 1.4– Concurrent GC for old generation - JDK 1.4 – Incremental Low Pause Collector (Train GC)
Parallel GC
• Parallel GC– Improve performance of GC– For young generation (Minor GC)– More than 4CPU and 256MB Physical
memory required
threads
timegc
threads
Default GC Parallel GC
Young Generation
Parallel GC
• Two Parallel Collectors– Low-pause : -XX:+UseParNewGC
• Near real-time or pause dependent application• Works with
– Mark and compact collector– Concurrent old area collector
– Throughput : -XX:+UseParallelGC• Enterprise or throughput oriented application• Works only with the mark and compact collector
Parallel GC
• Throughput Collector– –XX:+UseParallelGC– -XX:ParallelGCThreads=<desired number>– -XX:+UseAdaptiveSizePolicy
• Adaptive resizing of the young generation
Parallel GC
• Throughput Collector– AggressiveHeap
• Enabled By-XX:+AggresiveHeap• Inspect machine resources and attempts to set various parameters to
be optimal for long-running,memory-intensive jobs– Useful in more than 4 CPU machine, more than 256M– Useful in Server Application– Do not use with –ms and –mx
• Example) HP Itanium 1.4.2 java -XX:+ServerApp -XX:+AggresiveHeap -Xmn3400m -spec.jbb.JBBmain -propfile Test1
Concurrent GC
• Concurrent GC– Reduce pause time to collect
Old Generation– For old generation (Full GC)
– Enabled by -XX:+UseConcMarkSweepGC
threads
timegc
threads
Default GC Concurrent GC
OldGeneration
Incremental GC
• Incremental GC– Enabled by –XIncgc (from JDK 1.3)– Collect Old generation whenever collect young generation– Reduce pause time for collect old generation– Disadvantage
• More frequently young generation GC has occurred.• More resource is needed• Do not use with –XX:+UseParallelGC and –XX:+UseParNewGC
Incremental GC
• Incremental GC
Minor GC
After many time of Minor GC
Full GC
Minor GC
Minor GC
Old Generation is collected in Minor GC
Default GC Incremental GC
Young Generation
OldGeneration
Incremental GC
• Incremental GC-client –XX:+PrintGCDetails -Xincgc –ms32m –mx32m
[GC [DefNew: 540K->35K(576K), 0.0053557 secs][Train: 3495K->3493K(32128K), 0.0043531 secs] 4036K->3529K(32704K), 0.0099856 secs][GC [DefNew: 547K->64K(576K), 0.0048216 secs][Train: 3529K->3540K(32128K), 0.0058683 secs] 4041K->3604K(32704K), 0.0109779 secs][GC [DefNew: 575K->64K(576K), 0.0164904 secs] 4116K->3670K(32704K), 0.0169019 secs][GC [DefNew: 576K->64K(576K), 0.0057541 secs][Train: 3671K->3651K(32128K), 0.0051286 secs] 4182K->3715K(32704K), 0.0113042 secs][GC [DefNew: 575K->56K(576K), 0.0114559 secs] 4227K->3745K(32704K), 0.0191390 secs][Full GC [Train MSC: 3689K->3280K(32128K), 0.0909523 secs] 4038K->3378K(32704K), 0.0910213 secs][GC [DefNew: 502K->64K(576K), 0.0173220 secs][Train: 3329K->3329K(32128K), 0.0066279 secs] 3782K->3393K(32704K), 0.0325125 secs
Young Generation GC Old Generation GC in Minor GC TimeMinor GC
Full GC
Sun JVM 1.4.1 in Windows OS
Mark-compact Better throughput
Incremental GC(Train) Better Pause
Parallel GC Best Throughput
Concurrent GC Best Pause
Garbage Collection Measurement
• -verbosegc (All Platform)• -XX:+PrintGCDetails ( JDK 1.4)• -Xverbosegc (HP)
Garbage Collection Measurement
• -verbosegc
[GC 40549K->20909K(64768K), 0.0484179 secs][GC 41197K->21405K(64768K), 0.0411095 secs][GC 41693K->22995K(64768K), 0.0846190 secs][GC 43283K->23672K(64768K), 0.0492838 secs][Full GC 43960K->1749K(64768K), 0.1452965 secs][GC 22037K->2810K(64768K), 0.0310949 secs][GC 23098K->3657K(64768K), 0.0469624 secs][GC 23945K->4847K(64768K), 0.0580108 secs]
Full GC
Total Heap Size
GC Time
Heap size after GC
Heap size before GC
GC Log analysis using AWK script
• Awk script
BEGIN{ printf("Minor\tMajor\tAlive\tFree\n");}{ if( substr($0,1,4) == "[GC "){ split($0,array," "); printf("%s\t0.0\t",array[3])
split(array[2],barray,"K") before=barray[1] after=substr(barray[2],3) reclaim=before-after printf("%s\t%s\n",after,reclaim) }
if( substr($0,1,9) == "[Full GC "){ split($0,array," "); printf("0.0\t%s\t",array[4])
split(array[3],barray,"K") before = barray[1] after = substr(barray[2],3) reclaim = before - after printf("%s\t%s\n",after,reclaim) } next;}
% awk –f gc.awk gc.log
※ Usage
gc.awk
Minor Major Alive Freed0.0484179 0.0 20909 196400.0411095 0.0 21405 197920.0846190 0.0 22995 186980.0492838 0.0 23672 196110.0 0.1452965 1749 422110.0310949 0.0 2810 192270.0469624 0.0 3657 194410.0580108 0.0 4847 19098
gc.log
GC Log analysis using AWK script
< GC Time >
GC Log analysis using HPJtune
※ http://www.hp.com/products1/unix/java/java2/hpjtune/index.html
GC Log analysis using AWK script
< GC Amount >
Garbage Collection Tuning
• GC Tuning– Find Most Important factor
• Low pause? Or High performance?• Select appropriate GC model (New Model has risk!!)
– Select “server” or “client”– Find appropriate Heap size by reviewing GC log– Find ratio of young and old generation
Garbage Collection Tuning
• GC Tuning– Full GC Most important factor in GC tuning
• How frequently ? How long ?• Short and Frequently decrease old space• Long and Sometimes increase old space• Short and Sometimes decrease throughput by Load balancing
– Fix Heap size• Set “ms” and “mx” as same• Remove shrinking and growing overhead
– Don’t• Don’t make heap size bigger than physical memory (SWAP)• Don’t make new generation bigger than half the heap
Jmeter / Threads Histogram
Jmeter /Threads Group Histogram
Example
Example
2004-01-08 오후 7:14
2004-01-09 오전 8 시 전후
2004-01-09 오후 7 시 전후
금요일 업무시간
2004-01-10오전 10 시 전후
2004-01-10오후 6 시 전후
PEAK TIME52000~56000 sec9 시 ~ 1 시간 가량
Before TunedOld Area
Example
Peak Time 시에 Old GC 시간이 4~8 sec 로 이로 인한 Hang 현상 유발이 가능함
Before TunedGC Time
Example
12 일 03:38A12 일 05:58P13 일 07:18A13 일 09:38P14 일 11:58A15 일 01:18A15 일 03:38P16 일 05:58A16 일 07:18P17 일 08:38A17 일 10:58P
Weekend
Mon Office
Our
Tue Office
Our
Thur Office
Our
Fri Office
Our
After AP TunedGC Time
Example
12 일 03:38A12 일 05:58P13 일 07:18A13 일 09:38P14 일 11:58A15 일 01:18A15 일 03:38P16 일 05:58A16 일 07:18P17 일 08:38A17 일 10:58P
Weekend
Mon Office
Our
Tue Office
Our
Thur Office
Our
Fri Office
Our
Summary
JVM Tuning Summary
• Determine JVM performance goal• Gather statistics on your application• Select hotspot compiler• Tuning heap• Check threading model• Feedback
More TipsMore Tips
Thread dump
• Thread dump– Enabled by
• Unix “kill –3 [JAVA PID]”• Windows “Ctrl+Break”
– Snapshot of java application– Can profiling “hang-up”, and “slow-down”
Thread dump example
""
• Thread dump when slowdown in WAS
ExecuteThread: '232' for queue: 'default'" daemon prio=5 tid=0x573ca630 nid=0xd2c waiting for monitor entry [0x5cebf000..0x5cebfdb8] at java.util.Hashtable.get(Hashtable.java:314) at java.util.ListResourceBundle.handleGetObject(ListResourceBundle.java:122) at java.util.ResourceBundle.getObject(ResourceBundle.java:371) at java.util.ResourceBundle.getObject(ResourceBundle.java:374) at java.text.DateFormatSymbols.initializeData(DateFormatSymbols.java:483) at java.text.DateFormatSymbols.<init>(DateFormatSymbols.java:99) at java.text.SimpleDateFormat.<init>(SimpleDateFormat.java:275) at java.text.SimpleDateFormat.<init>(SimpleDateFormat.java:264) at XXX.uv.com.cm.CmDateTimeUtil.getCurrentTime(CmDateTimeUtil.java:88) at XXX.uv.com.util.CmLog.setFileLog(CmLog.java:171) at XXX.uv.com.jsp.EjbJspBase.service(EjbJspBase.java:371) at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:265) at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:200) at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:2546) at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2260) at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:139) at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)
"ExecuteThread: '231' for queue: 'default'" daemon prio=5 tid=0x573f9a60 nid=0x13a8 waiting for monitor entry [0x5ce7f000..0x5ce7fdb8] at java.util.Hashtable.get(Hashtable.java:314) at java.text.DecimalFormatSymbols.initialize(DecimalFormatSymbols.java:333) at java.text.DecimalFormatSymbols.<init>(DecimalFormatSymbols.java:55) at java.text.NumberFormat.getInstance(NumberFormat.java:565) at java.text.NumberFormat.getInstance(NumberFormat.java:324) at java.text.SimpleDateFormat.initialize(SimpleDateFormat.java:327) at java.text.SimpleDateFormat.<init>(SimpleDateFormat.java:276) at java.text.SimpleDateFormat.<init>(SimpleDateFormat.java:264) at XXX.uv.com.cm.CmDateTimeUtil.getCurrentTime(CmDateTimeUtil.java:88) at XXX.uv.com.cm.CmDateTimeUtil.getCurrentTime(CmDateTimeUtil.java:67) at XXX.uv.com.datastu.DateTime.setCurrentTime(DateTime.java:190) at XXX.uv.com.jsp.EjbJspBase.service(EjbJspBase.java:239) at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:265) at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:200) at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:2546) at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2260) at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:139) at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)
• Profiling CPU usage/HP UX– HP UX : Glance + Thread Dump
HP Glance
Press “G”
Thread monitoring
• Profiling CPU usage/HP UX
"Application Manager Thread" prio=8 tid=0x002a6c00 nid=62 lwp_id=15999 waiting on monitor [0x64bce000..0x64bce4b8] at java.lang.Thread.sleep(Native Method) at weblogic.management.mbeans.custom.ApplicationManager$ApplicationPoller.run(ApplicationManager.java:1137)
CPU Load of Thread 15999 is 17.7%
Thread 15999 is working on weblogic.management.mbeans.custom.ApplicationManager(ApplicationManger.java 1137)
Glance Thread Monitoring
Java Thread Dump
• Other tools– Profile with Java option– Analyze using HP Jmeter– Jprobe– Stress Test
• Load Runner• MS Stress (Free)
• Related URL– Java Thread http://java.sun.com/docs/hotspot/threads/threads.htm– Java Performance http://java.sun.com/docs/hotspot/PerformanceFAQ.html– Java Thread http://www.javaworld.com/javaworld/jw-09-1998/jw-09-threads.html– Pick up performance with generational gc
http://www.javaworld.com/javaworld/jw-01-2002/jw-0111-hotspotgc.html– JVM1.4 GC Tunning http://java.sun.com/docs/hotspot/gc1.4.2/index.html– HP Jmeter,Jtune,Jconfig http://www.hp.com/products1/unix/java/developers/index.html– SPECjvm98– SPECjAppServer2001/2002
Thank you
