Java GC - Pause tuning

Everything you ever wanted to know about GC pauses**but were afraid to ask

1

Death by pauses

Tuesday, July 10, 12

Agenda

1. Introduction

2. Crime Scene Investigation

3. JVM Memory management systems and tools

4. Putting it together

2Tuesday, July 10, 12

The Crime ScenePG 13*

* Parents strongly cautioned: typed language, dead objects and verbose logs may not be suitable to scripting language fans


4

Apa

che

Tom

cat

Ora

cle

B2Ce-commerce platform


4

Apa

che

Tom

cat

Ora

cle


•12+ Servers•10 different Webapps•50+ JVMs (Oracle JDK6)


4

Apa

che

Tom

cat

Ora

cle


•12+ Servers•10 different Webapps•50+ JVMs (Oracle JDK6)

•> 30000 sessions•250-400 Req/s•Variance is high


... an unusual victim...

5

Product catalog modeled as a Graph100% custom implementation

100% on-heap (no SQL except for initial load)in-place update by AtomicReference.set()


... an unusual victim...

5

Product catalog modeled as a Graph100% custom implementation

100% on-heap (no SQL except for initial load)in-place update by AtomicReference.set()

Caching aggressively is not possibleLarge number of request-scoped objects

Many WS into backoffice systems = latency


6

vs.

Throughput Latency


7

Interactive e-commerce app:Low latency is the top

priority!


The Crime Scene

8

Time

JDBC Connections


The Crime Scene

8

Time

JDBC Connections

Time

Requests/s


The Crime Scene

8

Time

JDBC Connections

Time

Requests/s

Time

Active threads


The Crime Scene

8

Time

JDBC Connections

Time

Requests/s

Time

Active threads

Time

HTTP Executor Queue Size


The evidence

9

Size

in M

B

1 hour

Heap


The evidence

9

Size

in M

B

1 hour

Heap

Can’t see anything: let’s zoom out!


The evidence

10

Size

in M

B

24 hours

Heap


0

25

50

75

100

1 hour

The evidence

11

Time spent in GC (%)

Size

in M

B

1 hour

Heap


The usual suspects...



• OutOfMemory Heap




• OutOfMemory PermGen





• Long GC pauses





• Long GC pauses

➡ under high load = immediate death





• Long GC pauses

➡ under high load = immediate death

12

Death by

pauses


Why do we need this GC thing again ?

13

“Many concurrent algorithms are very easy to write with a GC and totally hard (to down right impossible) using explicit free.”

Cliff Click


Fine, we just need to tune the JVM, right?...



14

POP QUIZZ!Number of command-line flags*?

* Oracle JVM 1.6.0_31 x86_64 server



14



less than 100 flags



14



less than 100 flags100 <= X< 200



14



less than 100 flags100 <= X< 200200 <= X< 300



14



less than 100 flags100 <= X< 200200 <= X< 300300 <= X< 400



14



less than 100 flags100 <= X< 200200 <= X< 300300 <= X< 400400 <= X< 500



14



less than 100 flags100 <= X< 200200 <= X< 300300 <= X< 400400 <= X< 500500 <= X< 600



14



less than 100 flags100 <= X< 200200 <= X< 300300 <= X< 400400 <= X< 500500 <= X< 600600 <= X< 700



14



less than 100 flags100 <= X< 200200 <= X< 300300 <= X< 400400 <= X< 500500 <= X< 600600 <= X< 700 664 Flags!


JVM

Memory in the JVM

16


JVM

17

Permanent (PermGen) Class metadatainterned Strings, etc.


Heap

18

Permanent (PermGen)

Application Objects

Class metadatainterned Strings, etc.


19

Young / New

Old / Tenured

Permanent (PermGen)


20

Old / Tenured

Eden S0 S1

Permanent (PermGen)


The Garbage Collector is generational

21

Old

Eden Sur

vivor 0

Surviv

or 1


22

Old

Eden Sur

vivor 0

Surviv

or 1

Allocation


23

Old

Eden Sur

vivor 0

Surviv

or 1


24

Old

Eden Sur

vivor 0

Surviv

or 1

100% = GC!


25

Old

Eden Sur

vivor 0

Surviv

or 1

LiveUnreferenced


26

Old

Eden Sur

vivor 0

Surviv

or 1

Copy


Reset...

27

Old

Eden Sur

vivor 0

Surviv

or 1


28

Old

Eden Sur

vivor 0

Surviv

or 1

Allocation


29

Old

Eden Sur

vivor 0

Surviv

or 1

100% = GC !


30

Old

Eden Sur

vivor 0

Surviv

or 1


31

Old

Eden Sur

vivor 0

Surviv

or 1

Copy


32

Old

Eden Sur

vivor 0

Surviv

or 1

Copy


33

Reset

Old

...

Eden Sur

vivor 0

Surviv

or 1

Génération 1

Génération 2


34

Old

Eden Sur

vivor 0

Surviv

or 1

Allocation


35

Old

Eden Sur

vivor 0

Surviv

or 1

100% = GC !


36

Old

Eden Sur

vivor 0

Surviv

or 1

Copy


37

Eden Sur

vivor 0

Surviv

or 1

Promotion

Old


Old


39

Old“Almost full” = GC !


42

Old

Compaction(optional)


Garbage Collectors

44

•Générational

• Stop the world!

• Throughput or Concurrent


GC characteristics

45

YoungYoung

OldOldOld

Serial Parallel

Serial

Parallel

Concurrent


GC characteristics

46

YoungYoung

OldOldOld

Serial Parallel

Serial Default

Parallel N/A

Concurrent


GC characteristics

47

YoungYoung

OldOldOld

Serial Parallel

Serial

Parallel

Concurrent


GC characteristics

47

YoungYoung

OldOldOld

Serial Parallel

Serial

Parallel

Concurrent

Serial


GC characteristics

47

YoungYoung

OldOldOld

Serial Parallel

Serial

Parallel

Concurrent

Serial Parallel


GC characteristics

47

YoungYoung

OldOldOld

Serial Parallel

Serial

Parallel

Concurrent

Serial Parallel

ParallelOld


GC characteristics

47

YoungYoung

OldOldOld

Serial Parallel

Serial

Parallel

Concurrent

Serial Parallel

ParallelOld

CMS


GC characteristics

47

YoungYoung

OldOldOld

Serial Parallel

Serial

Parallel

Concurrent

Serial Parallel

ParallelOld

CMSCMS Serial


GC characteristics

48

YoungYoung

OldOldOld

Serial Parallel

Serial Serial Parallel

Parallel ParallelOld

Concurrent CMS Serial CMS

Parallel implementation actually differ for each variant


GC characteristics


CMS is the right choice

50

Serial

Parallel

ParallelOld

CMS

CMS Serial

0 250 500 750 1000

937

871

846

852

917

Average test duration (s)


Tools: CLI

51

jps, jhat, jmap, jstack, jstat

$ jstat -gcutil PID S0 S1 E O P YGC YGCT FGC FGCT GCT 0.00 40.88 58.41 18.34 66.65 2729 316.538 46 6.820 323.358


Tools: GUIs


Tools: GUIs (2)

•Any profiler

•During development

• For autopsies!


Tools: GUIs (2)

•Any profiler

•During development

• For autopsies!

53

HeapDumpOnOutOfMemoryErrorHeapDumpPath


verbose:gc


verbose:gc

54

Stop the world!


MBeans


OK, so we can measure... temperature!!


OK, so we can measure... temperature !

57

=


58Credit: http://www.lhup.edu/mkhalequ/fieldtrip/geos253.htm

But...a single temperature measure is not enough to diagnose anything!

We must archive all measurementsto know the baseline!


http://www.lhup.edu/mkhalequ/fieldtrip/geos253.htm


Therefore we must persist all measurements!

• JMX + jmxtrans

• RRD

• Graphite

• etc.


Operating the (many) switches only makes sense...

60Credit: http://www.our-energy.com




...if we can measure/compare the effects!

61

Before

After

cput

ime


Putting it together


63

We want to minimize the GC pauses

Young (ParNew)Old (CMS-initial-mark + CMS-remark)


64

vs.


JVM

Tomcat

64

Application(code)

vs.


1. Code

• Tuning the JVM cannot compensate for bad code

• Rules of thumb

• Immutability = object reuse = less allocations *

• Move code invariants out of tight loops

• Know the characteristics of your data structures & frameworks (java.util, Guava, Hibernate, etc.)

• Mind the gap: data structure overhead can kill you!

65* But...pooling can be counter-productive!


Example : HashMap

66

HashMap

Entry[16]

Entry

value

key

48

80

32


Example : HashMap

66

HashMap

Entry[16]

Entry

value

key

48

80

32Overhead = 160 Bytes!


Example : HashMap

66

HashMap

Entry[16]

Entry

value

key

48

80

32Overhead = 160 Bytes!

•SingletonMap (40 Bytes)•initialCapacity + loadFactor


Less allocations...

67

GC

You

ng /

s


... saves CPU

68

Cha

rge

CPU


2. Tomcat

• Pooling

• JSP tags: enablePooling in web/webdefault.xml

• -Dorg.apache.jasper.runtime.JspFactoryImpl.USE_POOL=false

• Careful with buffers and their reuse

• -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true

• JSP usage is a factor in PermGen requirements

• Test & Measure, always!


2. Tomcat

• Pooling

• JSP tags: enablePooling in web/webdefault.xml

• -Dorg.apache.jasper.runtime.JspFactoryImpl.USE_POOL=false

• Careful with buffers and their reuse

• -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true

• JSP usage is a factor in PermGen requirements

• Test & Measure, always!

69

!Pooling may lead

to Old fragmentation!


3. The JVM

70

Hea

p Si

ze

Time

Hea

p Si

ze

Time


3. The JVM

70

Hea

p Si

ze

Time

Hea

p Si

ze

Time

pause > 1s !


3. The JVM

70

Hea

p Si

ze

Time

Hea

p Si

ze

Time

Frequent GC

pause > 1s !


The heap

71

Heap


The heap

71

Heap-Xms : start size-Xmx : max size


Young vs Old

72

Young

Old


Young vs Old

72

Young

Old

-XX:NewSize -XX:MaxNewSize-XX:NewRatio


Young vs Old

72

Young

Old•“Working Set”•Caches, object pools•HttpSession, average lifespan objects

Objects < RequestScope


First mistake: setting the Young too small

73

Young

Old


First mistake: setting the Young too small

73

Young

Old

Young fills up quickly = many GC Young

Objects promoted to Tenured too fast = many GC Old


Second mistake: setting Young too large

74

Young

Old


Second mistake: setting Young too large

74

Young

Old

GC Young pauses increase


Tuning Young

75

Young

Old

Default NewRatio=8 with -server on Intel

=Too small for a webapp with non-

trivial load!


Tuning Young

75

Young

Old

Increase Young slowly and measure the effects!


Old: Mind the Gaps (fragmentation)!

76

Young

Old


Old: Mind the Gaps (fragmentation)!

76

Young

OldJDK6 < u22:

ParNew (prom

otion failur

e size = 155

95) (promoti

on failed)


Old generation : ideal shape


Old generation : real life


Old generation : ideal vs. real


Old generation : ideal vs. real

79

Rate increases


Things to watch for

• Traffic/Load variance

• Traffic increases => Memory pressure increase

• CMS requires some headroom to operate properly

• Several phases are concurrent, i.e. at the same time as new objects are allocated


Things to watch for

• Traffic/Load variance

• Traffic increases => Memory pressure increase

• CMS requires some headroom to operate properly

• Several phases are concurrent, i.e. at the same time as new objects are allocated

80

(concurrent mode failure): 2165740K->1284261K(2228224K), 8.9411250 secs


Giving CMS some room to operate

81

Young

Old



81

Young

Old

CMSInitiatingOccupancyFraction = 92%

This is the default....


We really need 75-80%

UseCMSInitiatingOccupancyOnly to force the JVM to only consider this criteria


81

Young

Old


82

CMS initial-mark


83

CMS initial-mark (cumulative)


83


Median: -83%


83


Median: -83%

Top 99%: -79%


84

CMS remark


85

CMS remark (cumulative)


85

CMS remark (cumulative)

Top 90%: -56%


But...I still see pauses !

• RMI triggers explicit GC regularly

• Invokes System.gc()

• Explicit GC = Full GC (Serial) = 4-8s stop-the-world pause !

• DisableExplicitGC + CMSClassUnloadingEnabled

• ExplicitGCInvokesConcurrentAndUnloadsClasses


Complete GC comparison


• Survivors tuning (S0 & S1)

• Size, ratio vs. Eden, max generation

• G1

• Principles and operations are radically different!

• Other JVMs : JRockit, Azul, IBM

• Check tuning validity after every code change!

• Measure, measure, measure!

89

What’s next?


90

Questions ?


Java GC - Pause tuning

Technology

Transcript of Java GC - Pause tuning