Memory Kineticsfor Scale-up and Optimization

Kenny Yu

Session ID: CON5144

a.

A large heap (-Xmx) is not good enough --Excessive GC activitiesPremature tenuringAllocation failuresOutOfMemoryError

Problems

High throughputLow pause timeLow cost of GC

Objectives

Objectives explained: Good GC

Approaches

Generational distributionGC intervals ⇦ Eden + SRMinimal major GC ⇦ OC

Approaches

Capacity = ArrivalRate * HoldTime

Applicable to subsystemsEdenSurvivorOldGen

Regardless of distribution of retention time

Fundamentals: Little's Law

1. Optimize GC interval and memory region sizes

Approaches

a. The application is re-designed to polarize the generational distribution:Lighter sessionsLong-living caches

Generational hypothesis: enforced

Pop

ulat

ion

Age

1. 180335.787: [Full GC [PSYoungGen: 57985K->0K(1850304K)] [PSOldGen: 1945111K->1483937K(2097152K)] 2003097K->1483937K(3947456K) [PSPermGen: 263211K->263211K(329728K)], 10.0093850 secs] 180352.182: [GC [PSYoungGen: 1603456K->42582K(1873344K)] 3087393K->1526520K(3970496K), 0.3015250 secs]!un-optimized

Measure fixed OldGen Utilization size

1.OUfixed = OU on full GC = 1483937KSurvivor copy rate = 42582/0.3 = 142 M/sFull GC interval = (OC - OUF) / (ΔOU / Δt)We want to make this 24 hrs.

1. Too brief: excessive overflow to OldGen

Too long: large footprint + long pause

Determine young GC interval

1. To measure: gc-free response time (RT0) at low stress

To set: young gc interval (Tygc) > 4 * RT0

To measure: memory allocation rate under load

Determine Eden Capacity

13714.538: [GC 13714.538: [ParNew: 4399678K->33530K(4806016K), 0.0518020 secs] 5692328K->1326181K(6903168K), 0.0519700 secs] [Times: user=0.15 sys=0.00, real=0.05 secs]

13768.579: [GC 13768.579: [ParNew: 4402682K->27479K(4806016K), 0.0508480 secs] 5695333K->1320130K(6903168K), 0.0510040 secs] [Times: user=0.17 sys=0.00, real=0.05 secs]

Δm/Δt = (4 402 682 - 33 530) / (68 - 14) = 80 M/s

EC = Δm/Δt * Tygc = 80M *30 = 2400M

Determine Survivor Capacity

1. set: SC > 2 * max(SU)

1. In our example, SC = 200 M to handle 2x surges.

To measure: Survivor utilization on oversized SC

EU ∝ responses/s

SU ∝ responses/s

Fix Survivor Ratio

1. SurvivorRatio is independent of workload,It is a property of the application.

1. In our example, 2400M/200M = 12

To measure: overflow rate under load-- How fast OldGen utilization grows

Determine Promotion Rate

jstat -gc pid 10000 60S0C S1C S0U S1U EC EU OC OU YGC FGC436864.0 436864.0 41019.9 0.0 4369152.0 3511510.8 2097152.0 1319641.3 127 4436864.0 436864.0 0.0 33325.8 4369152.0 197903.8 2097152.0 1319645.8 128 4436864.0 436864.0 0.0 33325.8 4369152.0 971835.5 2097152.0 1319645.8 128 4 436864.0 436864.0 0.0 33325.8 4369152.0 2003800.7 2097152.0 1319645.8 128 4436864.0 436864.0 0.0 33325.8 4369152.0 3422374.2 2097152.0 1319645.8 128 4436864.0 436864.0 48272.6 0.0 4369152.0 1306482.6 2097152.0 1319646.9 129 4

PromoRate = ΔOU/Δt = (6.9-1.3)/50 = 0.11 k/s

With adequate full GC interval (1 day)

Determine Old Gen Capacity

1. SpareOC = PromoRate * 3600 * 24= 10 M

OC = OUfixed + SpareOC

Summary of approach

Condition Measure Determine

Upon full GC fixed OldGen util OldGen capacity

Low load response time YGC interval

Load to 50% CPU memory alloc rate Eden size

Oversized survivor

survivor utilization SurvivorRatio

Load, EC, SC promotion rate FGC interval

1. -XX:SurvivorRatio=10 -XX:-UsePSAdaptiveSurvivorSizePolicy -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSScavengeBeforeRemark -XX:CMSInitiatingOccupancyFraction=95 -XX:+UseCMSInitiatingOccupancyOnly -Xms7g -Xmx7g -Xmn5g

JVM Memory Configuration

1.Keep constants:Fixed OldGen utilizationSurvivor ratio

CPU speed-dependentYoung GC interval

Proportional to CPU and Throughput:Eden sizeSpare OC

Memory scale-up to faster and more CPUs

1. Full GC once at nightMemory problems resolvedGC pause time 50 ms Application became CPU-boundResponse time outliers eliminated -- stability

Performance Results

• DIRECTV dev team, led by Bob Hays, Ankush Walhekar and Harsha Anna, for the application and configuration, and for redesigning the application for object lifetime management. Jamal Haskins for timely assistance on build and deploy.

Owen Li, for helping with LoadRunner scripts

Louis Liao for tuning Linux OS

Rao Uppalapati, Prem Chaitanya and Abhiram Madadi, for setting up environments

DIRECTV for support and providing the equipment and funding the trip

Acknowledgments

• Thank you.

Questions?