Post on 16-Sep-2018
Copyright © 2014 Splunk Inc.
Damien Dallimore Dev Evangelist , CSO Office @ Splunk
Splunking the JVM
Disclaimer
2
During the course of this presentaGon, we may make forward-‐looking statements regarding future events or the expected performance of the company. We cauGon you that such statements reflect our current expectaGons and
esGmates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-‐looking statements,
please review our filings with the SEC. The forward-‐looking statements made in the this presentaGon are being made as of the Gme and date of its live presentaGon. If reviewed aSer its live presentaGon, this presentaGon may not contain current or accurate informaGon. We do not assume any obligaGon to update any forward-‐looking statements we may make. In addiGon, any informaGon about our roadmap outlines our general product direcGon and is subject to change at any Gme without noGce. It is for informaGonal purposes only, and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligaGon either to develop the features or funcGonality described or to
include any such feature or funcGonality in a future release.
7
JVM = Java Virtual Machine
• First appeared in the early 90’s • Now the dominant runGme for enterprise applicaGons
ApplicaGon Servers Enterprise Service Buses Databases
NoSQL Distributed Big Data Web Servers
Directory Servers Search Engines Build Systems
Gaming Pladorms Trading Systems ReservaGon Systems
Core Banking Messaging Infrastructure Proprietary Systems
Many JVM Variants
9
More or less the same. OpenJDK is reference impl
for Java 7/8
Gehng merged into Hotspot
Commercial alternaGves
Oracle Hotspot OpenJDK Oracle JRockit IBM J9 Azul Zing
JVMs are here to stay
10
• It’s not just about the “J” • Big Data frameworks • You should care about gehng insights into your JVM data
Data Sources
12
• Logs • Developers • JMX (Java Management Extensions) • InstrumentaGon Agents • OperaGng System • Distributed CommunicaGons
Standard Log Files
14
Oct 21, 2013 4:42:15 PM org.apache.catalina.startup.Catalina load INFO: IniGalizaGon processed in 1153 ms Oct 21, 2013 4:42:15 PM org.apache.catalina.core.StandardService startInternal INFO: StarGng service Catalina • ApplicaGon logs that are part of the product • Developer logs for any code that was deployed • Wrisen to local disk or network storage
Garbage Collector Logs
15
54.736: [Full GC 54.737: [Tenured: 172798K-‐>18092K(174784K), 2.3792658 secs] 257598K-‐>18092K(259584K), [Perm : 20476K-‐>20476K(20480K)], 2.4715398 secs] [Times: user=0.56 sys=0.05, real=0.07 secs] • Generated by way of arguments passed to the JVM at startup -‐verbose:gc -‐Xloggc:/home/damien/jvm_logs/gc.log -‐XX:+PrintGC -‐XX:+PrintGCTimeStamps -‐XX:+PrintGCDetails
SplunkJavaLogging
16
• SomeGmes you can’t write to file or deploy a UF • Appenders for Java UGl Logging , Log4J , Logback • Simply add a logging appender to your logging configuraGon file
Splunk SDK for Java
21
• Use the SDK from any JVM Language , Java / Groovy / Scala etc…. • Send log events via REST , UDP or TCP directly to Splunk from your
code • Spring IntegraGon Adaptors available on Github • SDK available from dev.splunk.com
25
JMX = Java Management Extensions
Monitor JVM via MBean asributes , operaGons and noGficaGons
JVM MBeans
Vendor MBeans
Custom Coded MBeans
Gehng this Data into Splunk
27
• Runs on all supported Splunk pladorms • Works with all main JVM variants • 100% Free and Open Source
36
In my experience , the data sources shown so far will be thorough enough for most of your JVM monitoring use cases
But what if you want to go deeper ?
SplunkJavaAgent
38
• Dynamic Byte Code InjecGon Agent for JVMs • No need to change any code to get APM level insights • This is all you pass to the JVM at startup :
-‐javaagent:splunkagent.jar
What Raw Metrics Can You Get ?
39
" Bytecode InjecGon – class loading – method execuGon – method Gmings – method call stack – caught & uncaught excepGons
" JMX – running in local in-‐memory mode
" Binary HPROF Memory dumps decoded in text – no more staGc hprof dumps and then loading into another Mem Analysis tool
The JVM Executes as a Process
45
Let’s see what the host OperaGng System can tell us : top
External programs that provide addiGonal JVM insights : jstat
Index this data in Splunk and correlate
Opportunity
53
Reuse some of the data collecGon approaches in this presentaGon and build out a specific JVM app • Publish it on apps.splunk.com - Make some money - Generate publicity for yourself / your company - Generate sales leads - Sell more Splunk licenses if you are also a partner - Do it because you love community collaboraGon - Become my beer buddy for life
Some JVM App Ideas
54
• Splunk for JBoss • Splunk for Tomcat • Splunk for Solr • Splunk for Cassandra
I am here to help you
Special Offer: Try Splunk MINT Express for Free! Splunk MINT offers a fast path to mobile intelligence. How fast?
Find out with a 6-‐month trial*
• Register for your free trial: hsp://mint.splunk.com/conf2014offer
• Download the Splunk MINT SDKs • Add the Splunk MINT line of SDK code and publish**
• Start gehng digital intelligence at your fingerGps!
*Offer valid for .conf2014 aIendees and coworkers of aIendees only.
**Trial allows monitoring of up to 750,000 monthly acQve users (MAUs).
56