Monitoring and analytics with was liberty

© 2015 IBM Corporation

Monitoring and Analytics with the WebSphere Liberty Profile

Don Bourne

Agenda

• Runtime Self-Checks

− Slow / Hung Request Detection

− Event Logs

• Performance Monitoring

– Monitor MXBeans

– Viewing Data in Jconsole

– Viewing Data in Admin Center

• Embedded Analytics

− Access Logs

− Message Logs / Binary Logs / FFDC

2

Runtime Self-Checks

Request Timing

Which requests are slow? Why are they slow?

4 NEW IN FEBRUARY 2015 LIBERTY BETA

5

Request Timing – Concepts Built on a new request probe framework

– Tracks REQUESTS made to the application server

• Requests have a unique request ID

• Requests have a tree of events – Builds a tree of EVENTS as each

request runs • Events have a type that identifies

what kind of event it is • Events have context info that

describes an instance of the event • Events have a duration

Servlet service

EJB method

JDBC query

JDBC update

EJB method

A single request with a tree of events

RequestID: AAC9KLwFFXT_AAAAAAAAAAN

NEW IN FEBRUARY 2015 LIBERTY BETA

6

Request Timing – What does it do? Slow requests Logs the request with the tree of events Logs the current stack for the thread the request is using Repeats 3 times

Hanging requests Logs the request with the tree of events Captures a series of 3 javacores

Noteworthy Does not attempt to abort requests Reports on work from a single thread Production-friendly Idle-friendly


7

Request Timing – Configuration <featureManager> <feature>requestTiming-1.0</feature> </featureManager>

<requestTiming slowRequestThreshold="10s" hungRequestThreshold="600s" includeContextInfo="true" sampleRate="1" <timing type="websphere.servlet.execute" slowRequestThreshold="20s" hungRequestThreshold="900s" /> /> NEW IN FEBRUARY 2015 LIBERTY BETA

DEMO

8

9

Request Timing - Output Sample slow request detection output: TRAS0112W: Request websphere.servlet.service|DayTrader Web | TradeScenarioServlet(AAC9KLwFFXT_AAAAAAAAAAN) has been running on thread 0000006b for 1549.460ms. The following stack trace shows what this thread is currently doing. <stack trace> The following table shows the events that have run during this request. Duration Operation 1552.012ms + websphere.servlet.service | DayTrader Web | TradeScenarioServlet 0.014ms websphere.session.getAttribute | R-ObCtcDfR8Zd9riQEMCh6R | uidBean 30.714ms websphere.servlet.service | DayTrader Web | TradeAppServlet 0.010ms websphere.session.getAttribute | R-ObCtcDfR8Zd9riQEMCh6R | uidBean 30.456ms websphere.servlet.service | DayTrader Web | /quote.jsp 28.903ms websphere.servlet.service | DayTrader Web | /displayQuote.jsp 0.194ms websphere.datasource.psExecuteQuery | jdbc/TradeDataSource | SELECT t0.CHANGE1, t0.COMPANYNAME, t0.H 1520.695ms + websphere.servlet.service | DayTrader Web | TradeAppServlet 0.013ms websphere.session.getAttribute | R-ObCtcDfR8Zd9riQEMCh6R | uidBean 0.190ms websphere.datasource.psExecuteQuery | jdbc/TradeDataSource | SELECT t0.ADDRESS, t0.CREDITCARD, t0.EMAIL, t0 0.135ms websphere.datasource.psExecuteQuery | jdbc/TradeDataSource | SELECT t0.ACCOUNTID, t0.BALANCE, t0.CREATIONDAT 0.110ms websphere.datasource.psExecuteQuery | jdbc/TradeDataSource | SELECT t0.CHANGE1, t0.COMPANYNAME, t0.HIGH, t0 0.299ms websphere.datasource.psExecuteQuery | jdbc/NoTxTradeDataSource | SELECT KEYVAL FROM KEYGENEJB WHERE KEYNAME 0.571ms websphere.datasource.psExecuteUpdate | jdbc/NoTxTradeDataSource | UPDATE KEYGENEJB SET KEYVAL = ? WHERE KEYN 0.126ms websphere.datasource.psExecuteQuery | jdbc/NoTxTradeDataSource | SELECT KEYVAL FROM KEYGENEJB WHERE KEYNAME 0.286ms websphere.datasource.psExecuteUpdate | jdbc/NoTxTradeDataSource | UPDATE KEYGENEJB SET KEYVAL = ? WHERE KEYN 522.450ms websphere.datasource.psExecuteUpdate | jdbc/TradeDataSource | INSERT INTO orderejb (ORDERID, COMPLETIONDATE, 230.333ms websphere.datasource.psExecuteUpdate | jdbc/TradeDataSource | INSERT INTO holdingejb (HOLDINGID, PURCHASEDAT 0.936ms websphere.datasource.psExecuteUpdate | jdbc/TradeDataSource | UPDATE accountejb SET BALANCE = ? WHERE ACCOUN


10

Request Timing - Output Sample hung request detection output: TRAS0114W: Request AAAg7irff7c_AAAAAAAAAAA has been running on thread 000000c9 for 600579.325ms. The following table shows the events that have run during this request. <event tree> CWWKE0067I: Java dump request received. JVMDUMP034I User requested Java dump using 'd:\wlp\usr\servers\s1\javacore.20150220.155505.13168.0001.txt' through com.ibm.jvm.Dump.JavaDump JVMDUMP010I Java dump written to d:\wlp\usr\servers\s1\javacore.20150220.155505.13168.0001.txt CWWKE0068I: Java dump created: d:\wlp\usr\servers\s1\javacore.20150220.155505.13168.0001.txt <1 minute passes> CWWKE0067I: Java dump request received. JVMDUMP034I User requested Java dump using 'd:\wlp\usr\servers\s1\javacore.20150220.155605.13168.0002.txt' through com.ibm.jvm.Dump.JavaDump JVMDUMP010I Java dump written to d:\wlp\usr\servers\s1\javacore.20150220.155605.13168.0002.txt CWWKE0068I: Java dump created: d:\wlp\usr\servers\s1\javacore.20150220.155605.13168.0002.txt <1 minute passes> CWWKE0067I: Java dump request received. JVMDUMP034I User requested Java dump using 'd:\wlp\usr\servers\s1\javacore.20150220.155705.13168.0003.txt' through com.ibm.jvm.Dump.JavaDump JVMDUMP010I Java dump written to d:\wlp\usr\servers\s1\javacore.20150220.155705.13168.0003.txt CWWKE0068I: Java dump created: d:\wlp\usr\servers\s1\javacore.20150220.155705.13168.0003.txt TRAS0115W: Request AAAg7irff7c_AAAAAAAAAAA, which was previously detected to be hung, has completed after 821583.742ms.


Event Logging Logs data at start and/or end of events

− Event type / Event context info1

− Event duration1

− Request ID1

Can be configured to

− Log at entry, exit, or both − Log only for specified list of event types − Log only for events longer than a certain duration (for logs written at event

exit only)

Log events will use log record extensions (filterable using binaryLog command) for − Type − Context info − Request ID

11

1Request and event details are provided by underlying Liberty Request Probe framework


Event Logging – Log Record Context

Key event logging fields are added to the LogRecordContext requestID type contextInfo

Fields from LogRecordContext are stored in log / trace records when binary logging is used

The wlp/bin/binaryLog command can filter for records by extension: binaryLog view myServer –-includeExtension="type=websphere.datasource.executeUpdate"

binaryLog view myServer –-includeExtension="type=websphere.datasource.executeUpdate" --format=advanced


Event Logging – Configuration

13

<featureManager> <feature>eventLogging-1.0</feature> </featureManager>

<eventLogging includeTypes="websphere.servlet.execute" minDuration="500ms" logMode="exit" sampleRate="1" includeContextInfo="true" />


DEMO

14

Event Logging Sample event logging output (basic format): Servlet: [6/18/14 16:21:35:761 IST] 0000002a EventLoggingProbeExtension.class I BEGIN requestID=AAADvUHkFwy-AAAAAAAAAAD # type=websphere.servlet.service # contextInfo=com.ibm.ws.request.timing.TestJDBC [6/18/14 16:22:04:643 IST] 0000002a EventLoggingProbeExtension.class I END requestID=AAADvUHkFwy-AAAAAAAAAAD # type=websphere.servlet.service # contextInfo=com.ibm.ws.request.timing.TestJDBC # duration=2.614ms

JDBC: [6/18/14 16:21:43:727 IST] 0000002a EventLoggingProbeExtension.class I BEGIN requestID=AAADvUHkFwy-AAAAAAAAAAD # type=websphere.datasource.executeUpdate # contextInfo=jdbc/exampleDS | create table cities (name varchar(50) not null, population int, county varchar(30)) [6/18/14 16:21:44:200 IST] 0000002a EventLoggingProbeExtension.class I END requestID=AAADvUHkFwy-AAAAAAAAAAD # type=websphere.datasource.executeUpdate # contextInfo=jdbc/exampleDS | create table cities (name varchar(50) not null, population int, county varchar(30)) # duration=0.231ms

Session: [6/18/14 16:22:04:655 IST] 0000002a EventLoggingProbeExtension.class I BEGIN requestID=AAADvUHkFwy-AAAAAAAAAAE # type=websphere.session.sessionReleased # contextInfo=default_host/jdbcTestPrj_3 [6/18/14 16:22:04:656 IST] 0000002a EventLoggingProbeExtension.class I END requestID=AAADvUHkFwy-AAAAAAAAAAE # type=websphere.session.sessionReleased # contextInfo=default_host/jdbcTestPrj_3 # duration=0.012ms [note that the contextInfo for servlets includes servlet name, pathinfo and query string when that info is present in the request]


Event Logging Zooming in...(JDBC example)

requestID=AAADvUHkFwy-AAAAAAAAAAD type=websphere.datasource.executeUpdate contextInfo=jdbc/exampleDS | create table cities (name varchar(50) not null, population int, county varchar(30)) duration=0.231ms 16

[6/18/14 16:21:44:200 IST] 0000002a EventLoggingProbeExtension.class I END requestID=AAADvUHkFwy-AAAAAAAAAAD # type=websphere.datasource.executeUpdate # contextInfo=jdbc/exampleDS | create table cities (name varchar(50) not null, population int, county varchar(30)) # duration=0.231ms


Performance Monitoring


Focus Areas JVM Performance : Analysis on JVM Performance Data , GC data Application Performance : Servlet Usage and response time, Sessions WAS Containers : Usages of ThreadPools, ConnectionPools

monitor-1.0 feature (introduced in WAS85 Liberty Profile) Collects data from various Liberty components Data reporting using MXBeans MXBean data can be viewed with JConsole1 or any standard JDK client Much of the data can also be viewed from Liberty admin center tools

18


JConsole (part of JDK) Graphical monitoring tool to monitor Java Virtual Machine (JVM) and java applications Comes as part of Java Development Kit (java/bin/jconsole) Connects to local or remote JVMs Interacts with JMX MBeans and MXBeans to get data or execute operations Provides a way to log counters into a CSV file

19

monitor-1.0 The monitor-1.0 feature monitors the following components:

− JVM JvmStats MXBean − Web applications ServletStats MXBean − Thread pool ThreadPoolStats MXBean − [8.5.5.0 or later] JAX-WS endpoints WebServiceStats MXBean − [8.5.5.0 or later] Session management SessionStats MXBean − [8.5.5.0 or later] ConnectionPool ConnectionPool MXBean

All MXBeans in a collective can be accessed by connecting to the collective controller using the JMX REST Connector and the RoutingContext MBean1

1see http://www-01.ibm.com/support/knowledgecenter/SSD28V_8.5.5/com.ibm.websphere.wlp.nd.doc/ae/rwlp_jmx_routing.html

20

monitor-1.0 JvmStats Sample Object Name: WebSphere:type=JvmStats

− Heap − FreeMemory − UsedMemory

21

− ProcessCPU − GcCount − GcTime − UpTime

monitor-1.0 ServletStats (1 MXBean per servlet) Sample Object Name: WebSphere:type=ServletStats,name=MyApp.MyServlet

− AppName − ServletName − RequestCount − ResponseTime (nanoseconds)

22

− Description − RequestCountDetails − ResponseTimeDetails

monitor-1.0 ThreadPoolStats Sample Object Name: WebSphere:type=ThreadPoolStats,name=Default Executor

− ActiveThreads − PoolSize − PoolName

23

monitor-1.0 SessionStats (1 MXBean per web application) Sample Object Name: WebSphere:type=SessionStats,name=default_host/trade_lite

− CreateCount − LiveCount − ActiveCount

24

− InvalidatedCount − InvalidatedCountbyTimeout

monitor-1.0 ConnectionPool (1 MXBean per connection pool) Sample Object Name: WebSphere:type=ConnectionPool,name=jdbc/TradeDataSource

− CreateCount − DestroyCount − ManagedConnectionCount

25

− WaitTime − ConnectionHandleCount − FreeConnectionCount

Monitoring Views in the Admin Center Explore Tool

26

monitor-1.0

Traditional PMI Support (for clients written to work with the full profile application server)

− monitoring data is available in PMI format (CountStatistics, RangeStatistics, etc.) via “Perf” Mbean “WebSphere:*,type=perf”

− Existing clients will be able to work (changes required in connectors)

Use JMX Rest Connectors to connect to Liberty Retrieve all MBeans Query for MBean type=Perf Invoke standard methods exposed by “Perf” MBean

27

monitor-1.0 - Configuration

28

<featureManager> <feature>monitor-1.0</feature> </featureManager> <monitor // Property to enable or disable Traditional PMI way of reporting. enableTraditionalPMI="false" // Allows user to enable/disable monitors based on group name such as // WebContainer,JVM,ThreadPool,Session,ConnectionPool,... filter="Session" />

monitor-1.0

Performance

Overhead is about 4%, when all components are monitored Overhead is lower when fewer components are monitored

29

Liberty Integrated Log Analytics

31

Liberty Integrated Log Analytics Data and analytics based on content

from logs / trace / ffdc / access logs

Liberty (user apps)

AnalyticsCollector-1.0

AdminCenter

AnalyticsEngine-1.0

messages.log

trace.log

access.log

binary log

ffdc files

Index

kiba

na.w

ar*

Admin Center Explore Tool

Kibana (not included with Liberty)


32

Integrated Log Analytics – Configuration

<featureManager> <feature>analyticsCollector-1.0</feature> <feature>analyticsEngine-1.0</feature> </featureManager>

<analyticsCollector logLevel="INFO" accessLogEnabled="true" ffdcEnabled="true" />

<httpEndpoint id="defaultHttpEndpoint" host="*" httpPort="9085" httpsPort="9448"> <accessLogging filepath="${server.output.dir}/logs/http_defaultEndpoint_access.log" logFormat="%h %u %t "%r" %s %b %D %{User-agent}i" /> </httpEndpoint>


DEMO

33

34

Integrated Log Analytics – Admin Center Views

Admin Center Explore Tool contains analytic views and monitoring data


35

Integrated Log Analytics – Sample Kibana Dashboard

We've experimented with deploying Kibana in the same server as our analytics engine (Kibana is not included with Liberty)


Useful Links – Full Profile

36

Using High Performance Extensible Logging to troubleshoot applications http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/index.jsp?topic=%2Fcom.ibm.websphere.nd.multiplatform.doc%2Fae%2Fttrb_usinghpel.html High Performance Extensible Logging (HPEL) http://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/index.jsp?topic=/com.ibm.iea.was_v8/was/8.0/ProblemDetermination/WASv8_HPEL/player.html System administration in WebSphere Application Server V8.5, Part 3 High Performance Extensible Logging (HPEL): http://www.ibm.com/developerworks/websphere/techjournal/1208_bourne/1208_bourne.html WebSphere Application Server v8.5 Cross Component Trace http://webspherecommunity.blogspot.ca/2012/07/websphere-application-server-v85-cross.html Use Cross Component Trace (XCT) request Ids to see which log entries were generated by each request http://www.youtube.com/watch?v=oXsfDoiNb5c Use the IBM WebSphere Cross Component Trace Logviewer to view WebSphere Application Server log files http://www.youtube.com/watch?v=qorwRZh4DgQ High Performance Extensible Logging and Cross Component Trace Tech Video https://www.youtube.com/watch?v=FKKfUdNux70 Configuring the memory leak policy http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/topic/com.ibm.websphere.nd.doc/ae/ttrb_configmemleak.html Memory leaks in Java Platform, Enterprise Edition applications http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/index.jsp?topic=%2Fcom.ibm.websphere.nd.doc%2Fae%2Fctrb_memleakdetection.html Configuring the hang detection policy (full profile) http://www-01.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.iseries.doc/ae/ttrb_confighangdet.html Using sensitive log and trace guard http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/topic/com.ibm.websphere.express.doc/ae/ctrb_sensitivelogtrace.html

Useful Links – Liberty Profile

37

Liberty Profile: Binary Logging http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/topic/com.ibm.websphere.wlp.express.doc/ae/cwlp_HPELOverview.html Binary logging for Liberty profile http://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/topic/com.ibm.iea.was_v8/was/8.5.5.0/content/WASV855_Binary_Logging/player.html?dmuid=20130807101913462846 Timed operations for Liberty http://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/topic/com.ibm.iea.was_v8/was/8.5.5.0/content/WASV855_Serviceability_TimedOperations/player.html?dmuid=20130815082326115285 Lab: Liberty profile timed operations for JDBC http://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/topic/com.ibm.iea.was_v8/was/8.5.5.0/content/WASv855_Liberty_Timed_Operations_Lab.pdf?dmuid=20130807102148029081 http://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/topic/com.ibm.iea.was_v8/was/8.5.5.0/content/WASv855Labs_libtops.zip?dmuid=20130807102201566285 Liberty Profile: Slow and hung request detection http://www-01.ibm.com/support/knowledgecenter/was_beta_liberty/com.ibm.websphere.wlp.nd.multiplatform.doc/ae/rwlp_requesttiming.html Liberty profile: Event Logging http://www-01.ibm.com/support/knowledgecenter/was_beta_liberty/com.ibm.websphere.wlp.nd.multiplatform.doc/ae/rwlp_eventlogging.html Liberty profile: Analytics http://www-01.ibm.com/support/knowledgecenter/was_beta_liberty/com.ibm.websphere.wlp.nd.multiplatform.doc/ae/rwlp_analytics.html Liberty monitoring feature http://www-01.ibm.com/support/knowledgecenter/SSD28V_8.5.5/com.ibm.websphere.wlp.nd.doc/ae/twlp_mon.html Creating remote JMX connections in Liberty https://developer.ibm.com/wasdev/docs/article_howto_remotejmx/ Configuring secure JMX connection to the Liberty profile http://pic.dhe.ibm.com/infocenter/wasinfo/v8r5/index.jsp?topic=%2Fcom.ibm.websphere.wlp.nd.doc%2Fae%2Ftwlp_admin_restconnector.html

Useful Links – Other

38

JVM Writing Verbose GC to a Specified Log -Xverbosegclog in AIX, Linux and Windows http://www-01.ibm.com/support/docview.wss?uid=swg21384096 PROBLEM DETERMINATION TOOLS IBM Support Assistant http://www.ibm.com/software/support/isa/ IBM Whole-system Analysis of Idle Time (WAIT) http://wait.ibm.com Memory Analyzer (MAT) https://eclipse.org/mat/ EDUCATION AND TRAINING IBM WebSphere Application Server V8.5.5 Problem Determination http://www-304.ibm.com/jct03001c/services/learning/ites.wss/ca/en?pageType=course_description&courseCode=WA591CE Support Technical Exchanges http://www.ibm.com/support/entry/portal/Scheduled_tech_exchanges/Software/WebSphere/WebSphere_brand_support_(general)

Notices and Disclaimers Copyright © 2015 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission from IBM.

U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM.

Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. THIS document is distributed "AS IS" without any warranty, either express or implied. In no event shall IBM be liable for any damage arising from the use of this information, including but not limited to, loss of data, business interruption, loss of profit or loss of opportunity. IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided.

Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice.

Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary.

References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business.

Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation.

It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law.

Notices and Disclaimers (con’t)

Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to interoperate with IBM’s products. IBM expressly disclaims all warranties, expressed or implied, including but not limited to, the implied warranties of merchantability and fitness for a particular purpose.

The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right.

• IBM, the IBM logo, ibm.com, Bluemix, Blueworks Live, CICS, Clearcase, DOORS®, Enterprise Document Management System™, Global Business Services ®, Global Technology Services ®, Information on Demand, ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, SoDA, SPSS, StoredIQ, Tivoli®, Trusteer®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.

http://www.ibm.com/legal/copytrade.shtml

Thank You

Your Feedback is Important!

Access the InterConnect 2015

Conference CONNECT Attendee Portal to complete your session surveys from your smartphone,

laptop or conference kiosk.

Monitoring and analytics with was liberty

Documents

Transcript of Monitoring and analytics with was liberty