JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage...
36
JVM Web Application Metrics & Monitoring FOLIO @krrrr38 2
Transcript of JVM Web Application Metrics & Monitoring · pull /push push server -(process)-> metrics storage...
JVM Web Application
Metrics & Monitoring FOLIO
@krrrr38
2
3
1.
2.
3. JVM Web Application
4
5
Metrics
6
Metrics
JVM
http
JMX
jcmd, jstat, ...
7
Metrics
server/process
pull /push
8
pull /pushpush
server -(process)-> metrics storage gateway
e.g. mackerel agent
pull
server <- metrics
e.g. prometheus
9
Example 1mackerel-agent (mackerel-jvm-plugin)
mackerel
https://mackerel.io/
process
10
jvm app (jmx, perf,...)
↓
jstat, jcmd
↓
mackerel-agent
↓
mackerel
↓
mackerel 11
Example 2prometheus, grafana
12
jvm app: /metrics prom metrics
↓
prometheus
↓
grafana
prometheus
13
Example 3telegraf
metrics interface gateway
telegraf
https://github.com/in�uxdata/telegraf
14
15
key value JMX
dropwizard metrics
16
key+labels value
17
response_time_GET_foo = 123msresponse_time_GET_bar = 250msresponse_time_POST_bar = 210msresponse_time_POST_foo_bar = 512ms
19
average/max/95%tile
response_time_GET_foo_average = 123msresponse_time_GET_foo_max = 123msresponse_time_GET_foo_95% = 123msresponse_time_GET_bar_average = 250msresponse_time_GET_bar_max = 250msresponse_time_GET_bar_95% = 250msresponse_time_POST_bar_average = 210msresponse_time_POST_bar_max = 210msresponse_time_POST_bar_95% = 210msresponse_time_POST_foo_bar_average = 512msresponse_time_POST_foo_bar_max = 512msresponse_time_POST_foo_bar_95% = 512ms
95%tile / ...?
20
SELECT * FROM metrics WHERE key IN (response_time_GET_foo_95%,response_time_GET_bar_95%,response_time_POST_bar_95%,response_time_POST_foo_bar_95%)
index
SELECT * FROM metrics WHERE key LIKE 'response_time_%_95\%'
21
response_time{"method"=GET, path=foo, percent=50} = 123msresponse_time{"method"=GET, path=foo, percent=95} = 123msresponse_time{"method"=GET, path=foo, percent=100} = 123ms
template
percent
method=POST
22
23
24
JVM Web ApplicationMetrics
25
heap usage
gc lifecycle
thread pool
connection pool
cpu
request count / response time
queue
...26
heap usage / gc lifecyclememory leak
Xmx
gc
( )
in-memory cache
512M~2G
FullGC STW
gc lifecycle
27
thread / connection poolblocking thread
pool active
e.g. servlet not nio api
jedis/jdbc/... thread pool in application
blocking
thread pool
thread pool
implicit global
default ForkjoinPool
28
server cpu / network / etcapplication
cpu
network
29
request count / responsetime / queue / ...
30
31
micrometer
32
micrometerJava
key+labels value
plugin
JVM System metrics, GC, logback, tomcat,
jetty, hikaricp
jmx, prom, in�ux, datadog agent,...
33
jmx, java agent, -XX:+FlightRecorder , ...
34
agentjava -jar
e.g. NewRelic
35
36
framegraph:
http://www.brendangregg.com/blog/2015-11-
06/java-mixed-mode-�ame-graphs.html
https://github.com/jvm-pro�ling-tools/perf-
map-agent
https://glowroot.org/
37
