Find Those Java Memory Leaks Fast

36
Find Those Java Memory Leaks Fast! Oracle JRockit Mission Control 4.0 Mark Prichard Java Platform Group, Oracle Fusion Middleware [email protected]

Transcript of Find Those Java Memory Leaks Fast

Find Those Java Memory Leaks Fast! Oracle JRockit Mission Control 4.0

Mark Prichard Java Platform Group, Oracle Fusion Middleware

[email protected]

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should

not be relied upon in making purchasing decisions.

The development, release, and timing of any features or functionality described for Oracle’s

products remains at the sole discretion of Oracle.

•  Fly-weight Java container for virtualized environments

•  Improve datacenter efficiency - do more with less

•  Simpler and more powerful VM management

•  Released Oracle WebLogic Suite Virtualization Option

JROCKIT VIRTUAL EDITION

•  High-performance real-time solution for standard Java

•  Industry leading Deterministic Garbage Collector

•  Millisecond response times with “five nines” guarantee

•  Improve application performance & latency with unique tooling

JROCKIT MISSION CONTROL

•  Complete insight into application & JVM behavior

•  Zero performance overhead in production environments

•  No application modification or configuration required

•  World-class performance •  Powerful diagnostics

•  Full support from Oracle

JROCKIT JVM

JROCKIT REAL TIME

JRockit Product Family

JRockit Mission Control •  Monitor health & performance in production •  A suite of powerful tools for

–  Performance Tuning –  Diagnostics –  Support

JRMC 4.0 Demos

•  Let’s look at some JRMC + Flight Recorder demo scenarios

Demo - Memleak •  Leaking char[] – We suspect there may be a problem with Leak

$DemoObject

Demo - Memleak •  (right-click) Add to Type Graph •  Expand backwards until the leak stops •  Hashtable$Entry looks problematic

Demo - Memleak •  (right-click on relation from Hashtable$Entry -> DemoObject) •  List Referring Instances to see instances pointing to DemoObject •  Hashtable$Entry instances listed in Instances View

Demo - Memleak •  (right-click) Expand to Root •  Shows DemoThread with field “table” referring to Hashtable

contining the leaking DemoObject

Demo – Latency Analyzer

•  Threads spending most of the time blocking each other

Demo – Latency Analyzer

•  Much better (removed synchronized logger calls)

Demo – GC Detailed Analysis

•  GC profile is bad – frequent (long) full collections •  Finalizers are the problem – avoid them where

possible

Demo – GC Detailed Analysis

•  Much better – simply by removing finalize() altogether

•  Finalizers can often interfere with GC

Demo – Code Analysis

•  Flight Recorder -> Code -> Hot Methods – LinkedList is inefficient here, better to use HashMap instead

Demo – Code Analysis

•  Flight Recorder -> Code -> Exceptions – Exceptions are expensive to process: abused by programmer here

JRMC 4.0 – What’s New?

•  JRockit Flight Recorder Support •  Historical Data Option & View •  WebLogic Triggers •  Memory Leak Advances

–  Same Look & Feel (SWT/JFace vs. Swing) –  Classloader support

•  Regular Expressions (query) •  Bug Fixes •  Eclipse Integration updated to Ganymede/Galileo

© 2010 Oracle Corporation Proprietary and Confidential 16

What is the JRockit Flight Recorder?

•  New in JRockit R28 •  “Circular buffer” in JRockit JVM that

stores diagnostic data –  Always on –  New data comes in and is stored, old

data dropped off •  Built-in integration with JRMC

–  Replaces JRMC Runtime Analyzer and Latency Analyzer

•  Very low/near zero overhead –  Uses data already used by JVM

•  Data can include events from the JVM and from any other event producer –  WebLogic Server (WLDF) –  Fusion Middleware (DMS)

Flig

ht R

ecor

ding

New Data

Old Data

Time

• What it is designed for? –  Provide diagnostic information in running production systems –  Capture most recent activity to enable analysis leading up to

an issue –  Capture data from all levels JVM, WLS, DMS, etc… –  Offline/offsite analysis can be done using the JRMC GUI –  JRockit dumps capture information to assist in crash-analysis

• What it is not designed for? –  Large event payloads or very high volumes of events –  Long history –  Not a replacement for Debug logging or the server logging

JRockit Flight Recorder Use Cases

JRockit Flight Recorder Summary

•  Short summary: JRA + LAT combined •  Collects diagnostic data, called events •  Continuous recording of events to RAM or Disk •  Low overhead, “always on” recording of basic events to

RAM possible •  There is a special Default profile. It has id=0, and is

designed to have an overhead around 1%. It writes to memory unless otherwise specified. It is off by default.

Flight Recorder will help you …

•  Solve JVM tuning issues •  Help you figure out why JRockit does

something “strange” •  Help you look “under the hood” of JRockit •  Help you look “under the hood” of software

running on top of JRockit, for example WLS

Flight Recorder Diagnostic Commands

•  You interact with Flight Recorder through Mission Control

•  Or with the jrcmd diagnostic commands: –  start_flightrecording –  stop_flightrecording –  check_flightrecording –  dump_flightrecording

start_flightrecording

•  jrcmd <pid> start_flightrecording –  name - Name of recording (string) •  settings - Settings file or preset (string) •  defaultrecording - Starts default recording (bool, false) •  delay - Delay recording start (time, 0s) •  duration - Duration of recording (time, 0s) •  filename - Resulting recording filename (string) •  compress - GZip-compress the resulting recording file (bool, false) •  maxage - Maximum age of buffer data (time, 0s) •  maxsize - Maximum size of buffers in bytes (long, 0)

•  To start the default recording: $> jrcmd 4711 start_flightrecording defaultrecording

•  Or start JRockit with the command: $> java -XX:FlightRecorderOptions=defaultrecording=true

start_flightrecording (cont.)

•  For example, to start a recording connecting to a running JRockit JVM with pid 4711, with the settings named “jrockit-se.jfs”, and with the recording named “Troubleshoot” outputting to the file “recording.jfr” use: $> jrcmd 4711 start_flightrecording name=Troubleshoot settings=jrockit-se.jfs filename=recording.jfr

dump_flightrecording

•  Stores a snapshot of a running recording •  Jrcmd <pid> dump_flightrecording

•  name - Recording name (string) •  recording - Recording id (long, -1) •  copy_to_file - Copy recording data to file (string) •  compress_copy - GZip-compress "copy_to_file" destination (bool, false)

•  Example: jrcmd 16749 dump_flightrecording recording=0 copy_to_file=cont.jfr

•  Will store a copy of recording with id=0 to a file named “cont.jfr”. •  Note: The recording ends up in the working directory of the JVM you

attach to! (This is probably not the directory from where you run 'jrcmd'.)

stop_flightrecording

•  jrcmd <pid> stop_flightrecording –  name - Recording name (string) –  recording - Recording id (long, -1) –  discard - Discards the recording data (bool, false) –  copy_to_file - Copy recording data to file (string) –  compress_copy - GZip-compress "copy_to_file" destination (bool, false)

•  Will stop the recording. Can save the stopped recording to a file.

WebLogic Events in JFR

•  Simple configuration: enabled and configured using a volume control per server: –  Off (default), Low, Medium, High

•  Each level has a predetermined data set –  Determined through consultation with dev groups and Support –  Data set was adjusted based on performance testing –  High volume includes user info and stack traces, which are

expensive data gathering operations

Instrumentation Extension

WLDF Instrumentation Configuration (Instrumented at Build Time) WS WebApp JDBC EJB Etc… JTA

WLDF Monitors and Action Groups

WLDF Instrumentation Engine

Extension Actions

Extension Data

WLDF Flight Recorder Actions

JRockit Flight Recorder

WLDF Flight Recorder Image Source

JRFR Producer API JRFR Consumer API

Diagnostic Image File With Flight Recorder Data

WLST, Console, WLDF Watch/Notification

WLST Image Manager

Extension supplied JRockit Supplied WLDF File/Artifact Available if JRockit is Used

Diagnostic Image File Without Flight Recorder Data

WLDF/JFR Integration Architecture

• What it is designed for? –  Provide diagnostic information in running production systems –  Capture most recent activity to enable analysis leading up to

an issue –  Capture data from all levels JVM, WLS, DMS, etc… –  Offline/offsite analysis can be done using the JRMC GUI –  JRockit dumps capture information to assist in crash-analysis

• What it is not designed for? –  Large event payloads or very high volumes of events –  Long history –  Not a replacement for Debug logging or the server logging

WLDF/JFR Integration Use Cases

WLS 10.3.3, JFR R28 and JRMC 4.0.1 “Quick Start Guide”

•  Turn on JFR default recording –  -XX:FlightRecorderOptions=defaultrecording=true –  Flight recorder command line documentation

•  Configure WLS to use Platform MBean Server –  Admin console: domain->configuration->general

tab (Advanced) –  Check Platform MBean Server Enabled (default in

10.3.3) –  Check Platform MBean Server Used (off by

default) –  NB: Platform MBean Server security –  -

Djavax.management.builder.initial=weblogic.mana

Working with JRockit Flight Recorder

•  Start JRMC, connect to WLS •  Configure WLS Diagnostic Volume (admin

console: server->configuration->general, “High”) •  Right mouse, select “Dump Default Recording”

JRMC + Flight Recorder

JRMC + Flight Recorder

JRMC + Flight Recorder

JRMC + Flight Recorder

Recommended Reading

Written by two senior members of JRockit Engineering team Highly Recommended Chapter 1 - Getting started Chapter 2 - Adaptive Code Generation Chapter 3 - Adaptive Memory Management Chapter 4 - Threads and synchronization Chapter 5 - Benchmarking and tuning Chapter 6 - JRockit Mission Control Chapter 7 - The Management Console Chapter 8 - The Runtime Analyzer Chapter 9 - The Flight Recorder Chapter 10 - The Memory Leak Detector Chapter 11 - JRCMD Chapter 12 - Using the JRockit Management APIs Chapter 13 - JRockit Virtual Edition Appendix A - Bibliography Appendix B - Glossary

36