Watching the Puppet Show

Puppet & SensuDelivering reliable services.

Sean Porter@PorterTech

+

FOCUS● The challenges

● DevOps & Infrastructure as Code

● Common pitfalls & failure cases

● Sensu

● Puppet & Sensu in practice

Let’s talk about softwareIt is eating the world.

“Software is eating the world”

- Marc Andreessen (2011)

SOFTWARE IS EATING THE WORLD!● Society has an insatiable hunger for software

○ It is becoming part of every facet of our lives

● Companies deliver value with software

● We need to deliver more software, better

software, faster, & reliably - Easy right?

Let’s talk about infrastructureA plethora of technologies.

NETFLIX 2013

“Set it all on fire, child”

- Overlord Manatee

Let’s talk about DevOpsWhat is DevOps?

WHAT IS DEVOPS?

“DevOps is continuously looking for new ways to break down silos, eliminate inefficiencies, and

remove the risks that prevent the rapid and reliable delivery of software based services”

- Damon Edwards, DevOps Cafe

WHAT IS DEVOPS?● Continuous improvement - there is no end

● Requires a culture that supports learning

○ Measurement - move the needles

○ Retrospectives (or blameless post-mortems)

● All about delivering better software, faster

Let’s talk about IaCWhat is Infrastructure as Code?

WHAT IS INFRASTRUCTURE AS CODE?

“Enable the reconstruction of the business from nothing but a source code repository, an

application data backup, and bare metal resources”

- Adam Jacob, Web Operations

WHAT IS INFRASTRUCTURE AS CODE?● It’s not just about reconstruction & repeatability

● IaC techniques scale effectively to manage large

numbers of hosts and services

● Apply & revert* changes quickly - move faster!

● All about delivering software, faster, & reliably

BASIC IaC WORKFLOW

It’s all software.

BASIC IaC WORKFLOW

No safeties.

INFRASTRUCTURE AS CODE● Break things at scale!

● Some changes cannot easily be undone

● System state & service health

● Coordinating with application deployments

● “Erosion” - Entropy

BASIC IaC WORKFLOW

BASIC IaC WORKFLOW

BASIC IaC WORKFLOW

Improve the feedback loop.

Provides continuous testing.

Let’s talk about SensuWhat is Sensu?

WHAT IS SENSU?● It’s a monitoring tool

○ Modern architecture

○ Uses service checks with a simple plugin spec

○ Defined inputs/outputs & very composable

○ Designed for IaC workflows

WHAT IS SENSU?● A global community

○ 300+ contributors

● Scalable, monitor tens of thousands of systems

● Commercially backed

○ Enterprise version (RBAC etc.)

○ Support, training, & professional services

WHAT PLATFORMS CAN SENSU MONITOR?● Fantastic multi-platform support!

● Linux (Debian, RHEL)

● Windows

● OS X

● FreeBSD

● Solaris (10, 11)

● AIX

July 11th, 2011

MODERN ARCHITECTURE● Designed for:

○ Dynamic infrastructure (EC2, Docker, etc.)

○ Public networks

○ Complex network topologies (hybrid cloud)

Automatic (de)registration of monitoring clients!

SERVICE CHECKS● Simple to write & understand

○ STDOUT & exit status code

● Provide context in multiple forms

○ Human readable messages

○ Formatted metrics (PerfData, Graphite, etc.)

● Placed top to bottom - service dependency chain

SENSU CLIENT SOCKET INPUTecho '{ \

"name": "mysql_backup", \

"output": "could not connect to mysql", \

"status": 2, \

"ttl": 90000 }' | nc localhost 3030

THE SENSU PIPELINE

PLUGINS & EXTENSIONS● github.com/sensu-plugins (checks, handlers, etc.)

● monitoring-plugins.org

● Many extensions to add protocols etc.

○ StatsD

○ InfluxDB

○ System Profile (metric collection)

JSON CONFIGURATION { "checks": { "mysql_replication": { "command": "check-mysql-replication.rb", "subscribers": ["mysql"], "interval": 30, "playbook": "http://wiki.example.com/mysql-replication-playbook" } }}

Puppet & SensuIn practice.

SENSU PUPPET MODULEforge.puppetlabs.com/sensu/sensu

● A module to install and configure Sensu● Well documented & tested (score ~ 5.0)● Types e.g. sensu_check_config● Awesome contributors! (101+)

○ jlambert121, jamtur01, rodjek, and more!

Let’s configure a Sensu serverSensu servers publish check requests and process

check results and events.

SENSU SERVERnode 'sensu-01.foo.com' { class { 'sensu': rabbitmq_host => 'rabbit.foo.com', rabbitmq_password => 's3cr3t',

redis_host => 'redis.foo.com',redis_password => 'p4s5w0rd',

server => true, api => true }

Let’s configure a Sensu clientOn an HTTP API host.

SENSU CLIENTnode 'api-01.foo.com' { class { 'sensu': rabbitmq_host => 'rabbit.foo.com', rabbitmq_password => 's3cr3t', subscriptions => [ 'production', 'api' ] } }

Let’s configure a Sensu handlerOn the Sensu server.

SENSU HANDLER CONFIGsensu::handler { 'slack': command => 'handler-slack.rb', timeout => 30, config => { 'webhook_url' => 'https://...', 'channel' => 'alerts', 'username' => 'sensu' }}

sensu::plugin { 'sensu-plugins-slack': type => 'package', pkg_provider => sensu_gem}

Let’s configure a checkRun an HTTP endpoint check on ALL API machines.

This check is configured on the Sensu server.

SENSU CHECK CONFIGsensu::check { 'api_http_response': command => 'check-http.rb -u https://127.0.0.1/health', interval => 20, subscribers => ['api'], aggregate => 'api_health', timeout => 60, handlers => ['slack']}

SENSU CHECK DEPENDENCIES

Install the check plugin on hosts expected to run it:

sensu::plugin { 'sensu-plugins-http': type => 'package', pkg_provider => sensu_gem}

Let’s configure a standalone check

Run an HTTP endpoint check on the local API machine.This check is configured on the API machine.

SENSU STANDALONE CHECK CONFIGsensu::check { 'api_http_response': command => 'check-http.rb -u https://127.0.0.1/health', interval => 20, standalone => true, aggregate => 'api_health', timeout => 60, handlers => ['slack']}

sensu::plugin { 'sensu-plugins-http': type => 'package', pkg_provider => sensu_gem}

SENSU IN OTHER PUPPET MODULES

Create a new class to be included:

e.g. apache/manifests/monitoring/sensu.pp

class apache::monitoring::sensu { sensu::check { 'apache-running': command => 'check-procs.rb -p /usr/sbin/httpd -w 100 -c 200 -C 1', handlers => ['slack'] }}

SENSU IN OTHER PUPPET MODULES

Add client subscriptions and custom attributes:

class apache::monitoring::sensu { sensu::subscription { 'apache': 'custom' => { 'ntp_server' => $ntp::servers[0], 'health_endpoint' => '/healthz' } }}

THE SENSU PIPELINE

PUPPET & SENSU

Let’s take it to the next levelPuppet module testing & Sensu.

SERVERSPEC RSpec tests for your servers:

describe service('httpd'), :if => os[:family] == 'redhat' do it { should be_enabled } it { should be_running }end

describe port(80) do it { should be_listening }end

RUNNING TESTS● Test Kitchen

○ github.com/neillturner/kitchen-puppet● Vagrant plugins

○ github.com/jvoorhis/vagrant-serverspec● Serverspec SSH● … choose your own adventure!

TEST ≈ MONITOR

TEST ≈ MONITOR

PUPPET MODULE TESTS AS SENSU CHECKS● Use the Sensu Serverspec check plugin

○ sensu-install -p serverspec

check-serverspec.rb \ -d /etc/sensu/serverspec -t '*_spec.rb'

SENSU SERVERSPEC CHECK CONFIGsensu::check { 'serverspec': command => 'check-serverspec.rb -d /etc/sensu/serverspec', interval => 30, standalone => true, timeout => 60, handlers => ['slack']}

sensu::plugin { 'sensu-plugins-serverspec': type => 'package', pkg_provider => sensu_gem}

SUMMARY● More software & infrastructure

● DevOps & IaC help us deliver software - faster!

○ No safeties!

● Monitoring MUST be part of the workflow

● Puppet & Sensu have a mutualistic relationship

sensuapp.orgSean Porter - @PorterTech

Questions?