Does Your Business Have Six Rearview Mirrors And No Windshield?
An Introduction to Rearview - Time Series Based Monitoring
-
Upload
victorops -
Category
Technology
-
view
406 -
download
1
Transcript of An Introduction to Rearview - Time Series Based Monitoring
![Page 1: An Introduction to Rearview - Time Series Based Monitoring](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a92a291a28abb8768b48e0/html5/thumbnails/1.jpg)
01.07.15
Rearview – time series based monitoring
![Page 2: An Introduction to Rearview - Time Series Based Monitoring](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a92a291a28abb8768b48e0/html5/thumbnails/2.jpg)
What's Rearview?
• “Passive” monitoring framework
• Polls time series database (TSD) at scheduled intervals
• Runs Ruby scripts against data
• Can send alerts to:– Email
– VictorOps
– PagerDuty (cough)
● Semi-secure
![Page 3: An Introduction to Rearview - Time Series Based Monitoring](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a92a291a28abb8768b48e0/html5/thumbnails/3.jpg)
Graphite
● TSD (think RRD tool for storage combined with Ganglia for visualization
● Rich HTTP API for querying/transforming metrics
● Incumbant (at LivingSocial)
● Does not scale well
● Wildcard queries hurt
● Hard to install
![Page 4: An Introduction to Rearview - Time Series Based Monitoring](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a92a291a28abb8768b48e0/html5/thumbnails/4.jpg)
Yeah, but why?
● “I want New Relic for business metrics” - Chad Fowler (VPE LivingSocial)
● “Measure all the things!”
● Started with cron-based scripts
● Lots of code duplication
● Had to edit Chef/Nagios to deploy
● Painful
● Enable ease of deployment
● Better dashboards
![Page 5: An Introduction to Rearview - Time Series Based Monitoring](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a92a291a28abb8768b48e0/html5/thumbnails/5.jpg)
System Overview
1) Rearview runs monitor at specified time
2) Fetches metric from Graphite
3) Runs Ruby script against data (optional)
4) Sends alert to alerting service or email (optional)
![Page 6: An Introduction to Rearview - Time Series Based Monitoring](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a92a291a28abb8768b48e0/html5/thumbnails/6.jpg)
Monitors
● Ruby function called with results of TSDB query
● Some helper methods and implicit vars in scope
● Raise exception with error message to indicate failure state
● Runs in exernal, forked Ruby process in SAFE mode. Semi-secure.
● Scripts have 5s to complete (configurable) or are killed
![Page 7: An Introduction to Rearview - Time Series Based Monitoring](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a92a291a28abb8768b48e0/html5/thumbnails/7.jpg)
Live demo! Lookout ...
![Page 8: An Introduction to Rearview - Time Series Based Monitoring](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a92a291a28abb8768b48e0/html5/thumbnails/8.jpg)
Related Software
● Riemann (riemann.io)
● Stream receiver
● Clojure monitors
● Great documentation
● More efficient (no polling)
● Very flexible
![Page 9: An Introduction to Rearview - Time Series Based Monitoring](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a92a291a28abb8768b48e0/html5/thumbnails/9.jpg)
Related Software (cont...)
● Grapfana (grafana.org)
● More configurable
● Supports
– Graphite
– InfluxDB
– OpenTSB
![Page 10: An Introduction to Rearview - Time Series Based Monitoring](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a92a291a28abb8768b48e0/html5/thumbnails/10.jpg)
Future directions
● Handle recovery notifications
● Support InfluxDB
– Modern, well supported
– Scalable
● Support streaming data
● Port to Rust, Nim?
● Name change. Ideas?
![Page 11: An Introduction to Rearview - Time Series Based Monitoring](https://reader034.fdocuments.us/reader034/viewer/2022042716/55a92a291a28abb8768b48e0/html5/thumbnails/11.jpg)
Resources
● Rearview - https://github.com/fooblahblah/rearview
● Graphite - http://graphite.readthedocs.org/en/latest/
● Grafana - http://grafana.org/
● InfluxDB - http://influxdb.com/
● Riemann - http://riemann.io/