Monitoring in the cloud with Puppet
-
Upload
kris-buytaert -
Category
Documents
-
view
5.868 -
download
1
description
Transcript of Monitoring in the cloud with Puppet
Monitoring in the CloudMonitoring in the Cloudwith Puppet
PuppetCamp Ghent 2013
Kris Buytaert
Kris BuytaertKris Buytaert● I used to be a Dev,I used to be a Dev,● Then Became an OpThen Became an Op● Chief Trolling Officer and Open Source Chief Trolling Officer and Open Source
Consultant @inuits.euConsultant @inuits.eu● Everything is an effing DNS ProblemEverything is an effing DNS Problem● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore● Some books, some papers, some blogsSome books, some papers, some blogs● Evangelizing devopsEvangelizing devops
Do you know what your children do at 5 am Do you know what your children do at 5 am in the morning ?in the morning ?
● Sleeping ?Sleeping ?
● Crashing a Party ?Crashing a Party ?
● Why are the cops at your front door ?Why are the cops at your front door ?
● What happened ?What happened ?
● How long have they been gone ?How long have they been gone ?
devopsdevops● CultureCulture
● (Lean)(Lean)
● AutomationAutomation
● MeasurementMeasurement
● SharingSharing
Damon Edwards and John WillisDamon Edwards and John Willis
Gene KimGene Kim
Monitoring is usually an Monitoring is usually an aftertoughtaftertought
ENOBUDGET, ENOTIMEENOBUDGET, ENOTIME
A sub movementA sub movement● #monitoringsucks trending#monitoringsucks trending
● https://github.com/monitoringsucks/https://github.com/monitoringsucks/
● 2008 Study :Nagios + Friends2008 Study :Nagios + Friends
● 2011 Conclusion : Nagios/Icinga are 2011 Conclusion : Nagios/Icinga are the only automatable alternatives the only automatable alternatives
#monitoringsucks#monitoringsucksMonitoring is AWESOME. Metrics are AWESOME. I love it. Here's what I don't love: Monitoring is AWESOME. Metrics are AWESOME. I love it. Here's what I don't love:
● Having my hands tied with the model of host and service bindings. Having my hands tied with the model of host and service bindings.
● Having to set up "fake" hosts just to group arbitrary metrics together Having to set up "fake" hosts just to group arbitrary metrics together
● Having to either collect metrics twice - once for alerting and another for trending Having to either collect metrics twice - once for alerting and another for trending
● Only being able to see my metrics in 5 minute intervals Only being able to see my metrics in 5 minute intervals
● Having to chose between shitty interface but great monitoring or shitty monitoring Having to chose between shitty interface but great monitoring or shitty monitoring but great interface but great interface
● Dealing with a monitoring system that thinks IT is the system of truth for my Dealing with a monitoring system that thinks IT is the system of truth for my environment environment
● Not actually having any real choicesNot actually having any real choices
John Vincent (@lusis) on his blog http://lusislog.blogspot.com/2011/06/why-John Vincent (@lusis) on his blog http://lusislog.blogspot.com/2011/06/why-monitoring-sucks.htmlmonitoring-sucks.html
Let's forget aboutLet's forget about● GroundWorkGroundWork
● ZenossZenoss
● ZabbixZabbix
● HypericHyperic
● CactiCacti
● MuninMunin
● BigBrotherBigBrother
What's different in the cloud ?What's different in the cloud ?
● ScaleScale
● VelocityVelocity
● ChangeChange
ChallengesChallenges● ReproducabilityReproducability
● SpeedSpeed
● AuditingAuditing
● Keeping stuff in sync Keeping stuff in sync
Thank Thank godgod Luke for Puppet Luke for Puppet
And Mark for CfengineAnd Mark for Cfengine
}}#monitoringsucks #monitoringsucks 20132013
What we wantWhat we want
● Small , wel suited componentsSmall , wel suited components
• CollectCollect
• Transport / MangleTransport / Mangle
• Analyse / ActAnalyse / Act
• VisualizeVisualize
CollectdCollectd● CollectsCollects
● Zillion PluginsZillion Plugins
• Nginx,apache, mysql, diskNginx,apache, mysql, disk
● Graphite Carbon PluginGraphite Carbon Plugin
● Send metrics to graphiteSend metrics to graphite
● https://github.com/KrisBuytaert/pupphttps://github.com/KrisBuytaert/puppet-collectdet-collectd
● Collect from Collect from anywhereanywhere
● FilterFilter
● Send anywhereSend anywhere
● Queuing supportQueuing support
Puppet and LogstashPuppet and Logstash● https://github.com/KrisBuytaert/vagrant-puppet-logstashhttps://github.com/KrisBuytaert/vagrant-puppet-logstash
● Includes Logstash / Kibana / Grok / ES modules as Includes Logstash / Kibana / Grok / ES modules as submodules ! submodules !
git clonegit clone
● git submodule init git submodule init
● git submodule update git submodule update
● vagrant up vagrant up
Graphite Graphite ● Graphing at ScaleGraphing at Scale
● Graphing at EaseGraphing at Ease
● Any metric is a graphAny metric is a graph
● echo "somestring $somevalue $timestamp" | echo "somestring $somevalue $timestamp" | nc <%= graphitehost %> 2003nc <%= graphitehost %> 2003
Graphite ComposerGraphite Composer
x
Graphite APIGraphite API
Gdash In actionGdash In action
Puppet and GraphitePuppet and Graphite● https://github.com/KrisBuytaert/vagrant-graphite/https://github.com/KrisBuytaert/vagrant-graphite/
● Includes Graphite / Gdash / Jmxtrans / Logster / Collectd / Includes Graphite / Gdash / Jmxtrans / Logster / Collectd / Statsd / Tattle and more modules as submodules ! Statsd / Tattle and more modules as submodules !
● git clonegit clone
● git submodule init git submodule init
● git submodule update git submodule update
● vagrant up vagrant up
Collectd & GraphiteCollectd & Graphite
Configuring a ServiceConfiguring a Service● PackagePackage
● ConfigConfig
● Service Service
<- we pray status is correct<- we pray status is correct
Apache Example:Apache Example:
Definition of Done:Definition of Done:
Done means : being monitoredDone means : being monitored
Icinga ?Icinga ?• Isn't nagios dead ?Isn't nagios dead ?
• Vibrant CommunityVibrant Community
• Throw great parties in NurnbergThrow great parties in Nurnberg
• Nobody can pronounce it anyhowNobody can pronounce it anyhow
• https://github.com/Inuits/puppet-icinga/https://github.com/Inuits/puppet-icinga/
Monitoring BaselineMonitoring Baseline● Deploy a host,Deploy a host,
● Add it to the monitoringAdd it to the monitoring
● Add collection toolsAdd collection tools
● Add check definitionsAdd check definitions
● Update the monitoring tool configUpdate the monitoring tool config
Stored ConfigsStored Configs
Collection and ExportCollection and Export
Export :Export :
@@resource { @@resource {
... }... }
Collect:Collect:
Resource <<| Resource <<| query |>>query |>>
Clean out nodes that dissapearClean out nodes that dissapear
puppet node clean puppet node clean
Exporting and Collecting Exporting and Collecting
Monitoring a VhostMonitoring a Vhost
It is not in production if it is not It is not in production if it is not monitoredmonitored
Triggers on GraphsTriggers on Graphs● Export Java MetricsExport Java Metrics
● JMXTransJMXTrans
● Export JMXConfigsExport JMXConfigs
● Configure NRPE CheckConfigure NRPE Check
● Export NagiosCheckExport NagiosCheck
● Collect JMX Exports on Collect JMX Exports on JMXTransNodeJMXTransNode
● Graph EmGraph Em
Collect Nagios Configs Collect Nagios Configs on Nagios Serveron Nagios Server
Triggers on GraphsTriggers on Graphs
Triggers on GraphsTriggers on Graphs
Conclusion:Conclusion:● Reproducable monitoring setupReproducable monitoring setup
● Dynamically generated monitoring Dynamically generated monitoring configconfig
● Code is available at Code is available at github.com/KrisBuytaert/snippetsgithub.com/KrisBuytaert/snippets
But,But,(pronounced in Ozzy dialect)(pronounced in Ozzy dialect)
htt
p:/
/ww
w.k
art
ar.
net/
201
3/0
1/m
onit
ori
ng
-suck
s---
a-r
ant/
#monitoringlove#monitoringlove
HomeworkHomeworkSensuSensu
ExtrememonExtrememon
......
RiemannRiemann
EsperEsper
[email protected]@inuits.eu
Further ReadingFurther Reading@krisbuytaert @krisbuytaert http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/http://www.inuits.eu/http://www.inuits.eu/
InuitsInuits
Duboistraat 50Duboistraat 502060 Antwerpen2060 AntwerpenBelgiumBelgium891.514.231891.514.231
+32 475 961221+32 475 961221
http://loadays.orghttp://loadays.org