Listen to the sounds of your application
64
Listen to the sounds of your application Listen to the sounds of your application
-
Upload
krzysztof-ciesielski -
Category
Technology
-
view
305 -
download
1
description
There are lots of valuable signals in our systems that we can capture and measure. Aside from monitoring physical resources and latencies, we can bring visibility to business data that we care about. We will present how to create a monitoring and alerting stack which gives a uniform view of those signals. Given such setup we can detect anomalies or system failures or examine how our application usage changes and plan our capacity in advance. During this 30 minute talk we will try to convince you – the Java developer – why you should care about applications metrics and logging from early on in development.
Transcript of Listen to the sounds of your application
Listen to the sounds of your application
Listen to the sounds of your application
!2
Krzysztof Ciesielski
@kpciesielski
softwaremill.com
Lublin Software Craftsmen
!4
• Monitoring, huh?
• Introducing Graphite
• Log analysis – the whys
• Logstash architecture & use cases
• Exploring logs with Kibana
Agenda
!5
Monitoring, huh?
!7
Types of measurements
!7
Types of measurements
Network
!7
Types of measurements
Network
Machine
!7
Types of measurements
Network
Machine
Application
!9
measurement > prediction
!10
!11
measurement >> prediction
!12
Our stack
!12
Our stack
Server
App
Server
App
…
!12
Our stack
Server
App
Server
App
…
Graphite Logstash
!12
Our stack
Server
App
Server
App
…
Graphite Logstash
!12
Our stack
Server
App
Server
App
…
Graphite Logstash
!12
Our stack
Server
App
Server
App
…
Graphite Logstash
!12
Our stack
Server
App
Server
App
…
Graphite Logstash
!12
Our stack
Server
App
Server
App
…
Graphite Logstash
Yammer Metrics
!13
Introducing Graphite
!14
http://graphite/render?target=server.web1.load&height=800&width=600
!15
Graphite
!16
PORT=2003 SERVER=graphite.your.org echo "local.random.diceroll 4 `date +%s`" | nc ${SERVER} ${PORT};
!17
successful.login.attempt 1 1384471287 successful.login.attempt 1 1384471297
!17
successful.login.attempt 1 1384471287 successful.login.attempt 1 1384471297
=> successful.login.attempt = 1
!18
Yammer Metrics
!18
Yammer Metrics
private final Meter successfulLogins = metrics.meter(name(LoginHandler.class, "successful")); !public void login(String user, String password) { if (canLogin(user,password)) { successfulLogins.mark(); // ... } else { // ... } }
!19
Metrics
!19
MetricsTypes:
!19
MetricsTypes:
•Gauges
!19
MetricsTypes:
•Gauges
•Counters
!19
MetricsTypes:
•Gauges
•Counters
•Meters
!19
MetricsTypes:
•Gauges
•Counters
•Meters
•Histograms
!19
MetricsTypes:
•Gauges
•Counters
•Meters
•Histograms
•Timers
!19
MetricsTypes:
•Gauges
•Counters
•Meters
•Histograms
•Timers
•Health Checks
!19
MetricsTypes:
•Gauges
•Counters
•Meters
•Histograms
•Timers
•Health Checks
Reporters:
•STDOUT
•CSV
•SLF4J
•JMX
•Graphite
•Ganglia
!20
Counters, meters… vs.
Gauges
!21
Aggregation
!23
You can also check out• collectd https://collectd.org
• StatsD https://github.com/etsy/statsd/
• Riemman http://riemann.io/
• Twitter’s Ostrich https://github.com/twitter/ostrich
• Ganglia http://ganglia.sourceforge.net/
• Dashboards: http://shopify.github.io/dashing/ https://github.com/obfuscurity/descarteshttps://github.com/obfuscurity/dusk
!24
Event Sourcing
Tracing and storing all the events
Current app state: replaying the stream
Great for data mining and analysis
!25
Log aggregation
Less invasive way to build an “event stream”
Logs are data with plenty of value
!26
Log aggregation
Log source
Log source
Log source
Central storage
Query engine
Web view
!27
Logstash
Open Source (written in JRuby)
!27
Logstash
Open Source (written in JRuby)
Gathers logs from various inputs
!27
Logstash
Open Source (written in JRuby)
Gathers logs from various inputs
Parses and extracts metadata
!27
Logstash
Open Source (written in JRuby)
Gathers logs from various inputs
Writes to various outputs
Parses and extracts metadata
!27
Logstash
Similar tools: !Splunk, Graylog, Fluentd, log.io
Open Source (written in JRuby)
Gathers logs from various inputs
Writes to various outputs
Parses and extracts metadata
!28
Logstash centralized architecture
!29
Logstash grok
2013-11-10 19:41:25.321 [main] INFO o.a.camel.impl.DefaultCamelContext !- Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds
!29
Logstash grok
2013-11-10 19:41:25.321 [main] INFO o.a.camel.impl.DefaultCamelContext !- Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds
time 2013-11-10 19:41:25.321
thread main
loglevel INFO
source o.a.camel.impl.DefaultCamelContext
message Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds
!30
Logstash simple use cases
Output: e-mail alert
ExceptionsSecurity issuesRare and business-meaningful events
Grok:
!31
Kibana
!32
Kibana checking general app activity
!33
Kibana Spotting anomalies
!34
Kibana Checking regular events
!35
Kibana How often is this feature used?
!36
Kibana Displaying multiple query results
Query: exception Type: applog
Query: index Type: mongolog
!37
References• Graphite http://graphite.readthedocs.org/en/latest/
• Yammer Metrics http://metrics.codahale.com/
• Logstash http://logstash.net/
• Surfing the event stream by Sam Newman at Geecon http://www.slideshare.net/spnewman/surfing-the-event-stream
• Lessons from Building and Scaling LinkedIn by Jay Krepshttp://www.infoq.com/presentations/linkedin-architecture-stack
• Code as Craft http://codeascraft.com/
!38
