monitoring configuration,

release & configuration management

Christoph Oelmüller

<christoph.oelmueller@epost-dev.de>

*

*

*say hello

*monitoring design considerations

*monitoring configuration management

*the problem

*two approaches to solve the problem

*1:n approach

*n:m approach

*a quick n:m approach howto

*summary: 1:n versus n:m

*

*physicist

*2008 - 2013: linux consultant

*2014 : operations engineer at EPost-

Development GmbH

*linux infrastructure, identity management,

configuration management, monitoring...

*

*

*requirement analysis

*technical requirements

* revision controlled configurations

* rollback

* automization

*organizational requirements

* feasable workflow

* keep it simple stupid (KISS)

*

*infrastructural requirements

*design your staging

* complete?

* identical?

*

*time & material & risk

*cost of a false alarm

*cost of a grumpy admin

devmonitor

tstmonitor

prdmonitor

*

separate your stages!

*

what to configure?

*

*hosts

*add host to monitoring

master

*services

*add service to

monitoring master

*deploy check/plugin

on node

dev-mastertst-

masterprd-

master

dev-nodetst-

nodeprd-node

*

*

*all stages are identical

*all stages are complete

*one monitoring master/stage

*call it 1:n approach

*1 master

*n nodes

CODE YOUR CONFIG

CLONE YOUR CONFIG

*

*stages aren‘t identical

*stages are incomplete

*stages are only implemented via different

monitoring masters

*call it n:m approach

*n master

*m nodes

*

dev-master

dev-node 1

dev-node 2

tst-master

tst-node 1

tst-node 2

prd-master

prd-node 1

prd-node 2

nodes

dev-master

tst-master

prd-master

*

*no staging related false alarms

*checks can‘t go amok in production

*configuration is tested properly in prd

*we‘ve talked about this...

*puppet & icinga, exported resources

monitoring configuration is completely described bypuppet

*

*stages are only reflected in monitoring masters

*non-prod doesn‘t alarm

*agent configuration is done via

puppet/package/however

*master configuration is done natively

*KISS

CLONE YOUR CONFIG!

*a quick howto

*

1. transfer prd stage to dev stage (files...)

2. edit configuration files/generate files (check_MK!)

3. sync files to VCS repository

* add/remove/modify

4. merge/copy dev-repo to prd-repo

* svn copy/svn merge/git branching

5. sync prd-repo to prd-filesystem

* again: add/remove/modify

workflow is done on dev-master

*

*

*local operations (on dev-master)

*checkout VCS

*checkin VCS

* remote operations (on prd-master)

*checkout VCS

*VCS operations

*transfer repositories (svn merge/git merge/...)

*operations are wrapped

*directly called for local operations

*puppet exec resources for remote operations

*

*checkout VCS

*from which repository?

* use a fact „stage“!

* a monitoring master can only checkout/sync its own stage

repository

*transfer repositories

*from where to where? the workflow knows...

*puppet runs on remote hosts are triggered by

mCollective

*addressing is done via „stage“ fact

*

*remote execution framework

*a simple ssh loop won‘t do it...

*message subscribe architecture

*parallel job execution

*a job => mCollective plugin

*ping, service, puppet, your own plugin

*watch out

*mco agent = subscriber

*mco client = message sender

*

*each directory is a repository

*??? => monitoring configuration is spread over the

filesystem

*ask cmk --paths

*the assignment dir to repo is configured in

„locations.cfg“

*configuration directories are working copies

*

file{‘/etc/mon/locations.cfg‘:

ensure => present,

source => puppet:///...,

notify => Exec[‘mon_checkout‘]

}

exec{‘mon_checkout‘:

command => ‘/usr/local/bin/mon checkout‘,

notify => Service[‘nagios‘],

}

service{‘nagios‘:

ensure => running,

}

*

*dirty little secrets

*only complete directories

*puppet run feedback?

*file permissions

*distributed/simultaneous work

*1:n versus n:m – the battle...

1:n n:m

separation complete separation partial separation

transfer

dev2prd

via release management

tools

on request

puppetization

level required

high low

risk low ...a bit more

complexity

(KISS)

high low

configuration via puppet natively

*

*risk versus implementation effort

risk

implementation effort

1:n

n:m

*

mailto:christoph.oelmueller@epost-dev.de

http://slideshare.net/christophoelmuller/