Orchestrate  Event-­‐Driven  Infrastructure  with  SaltStack

Love  Nyberg  Site  reliability  &  Automa@on  Engineer

SaltStackSaltStack  is  extremely  fast  and  scalable  systems  and  configura@on  management  soDware  for  predic@ve  

orchestra@on,  cloud  and  data  center  automa@on,  server  provisioning,  applica@on  deployment  and  more.

Agenda  salt  @  bloglovin    Terminology    Orchestrate    Reactor    Beacons

salt  @  bloglovintens  of  millions  of  monthly  unique  visitors  hundred  of  millions  requests  per  day  couple  of  million  stories  parsed  per  day  

hybrid  bare  metal  servers/cloud  vm’s  2  salt  masters  ~90  salt  minions

salt  @  bloglovinParts  we  use!  

States/Pillars/Grains  Orchestrate  Reactor  Salt  Api  Salt  Cloud

TerminologyConfigura)on  Master/Minion  Masterless  

Transport  ZeroMQ  Raet  SSH  

Parts  States  Modules  Grains  Pillars  

Execu@on

salt '<target>' <function> [arguments]

Execu@on  modulesroot@master:/# salt minion_love network.interface eth0 minion_love: |_ ---------- address: 172.17.0.9 broadcast: None label: eth0 netmask: 255.255.0.0

Execu@on  modulesroot@master:~# salt minion1 cmd.run "ip addr show eth0" minion1: 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc pfifo_fast state UP group default qlen 1000 link/ether 12:4b:af:05:be:cd brd ff:ff:ff:ff:ff:ff inet 10.0.0.206/24 brd 10.0.0.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::104b:afff:fe05:becd/64 scope link valid_lft forever preferred_lft forever

State/srv/salt/state/sshd/init.sls

ssh: pkg.latest - name: openssh-server

salt test_minion state.sls ssh

State/Execu@on/srv/salt/state/sshd/init.sls ssh: pkg.latest - name: openssh-server

root@master:/# salt test_minion pkg.latest_version openssh-server

{%- set ips = salt['network.ip_addrs'](None, False, '172.17.0.0/24') %}

ssh: pkg.latest - name: openssh-server service.running: - enable: True - watch: - file: ssh - pkg: ssh file.managed: - name: /etc/ssh/sshd_config - source: salt://sshd/files/sshd_config - template: jinja - context: ips: {{ ips }}

Grainsvia state

root@mon1:/# salt minion_love grains.get ip_interfaces:eth0 minion_love: - 172.17.0.9

within a state file {%- set eth0 = salt['grains.get'](ip_interfaces:eth0, '') %}

Pillarroot@master:/# salt minion_love pillar.item secrets minion_love: ---------- secrets: ---------- foo: bar

Apply  state

root@master:/# salt minion_love state.highstate

salt '<target>' <function> [arguments]

Event  system

Event  system  is  a  local  ZeroMQ  PUB  interface  which  fires  salt  events  

Every  event  has  a  tag  and  a  data  structure

Event  chain  `test.ping`Event fired at Wed May 27 22:00:14 2015 ************************* Tag: new_job Data: {'_stamp': '2015-05-27T20:00:14.774445', 'arg': [], 'fun': 'test.ping', 'jid': '20150527220014773556', 'minions': ['minion_love'], 'tgt': 'minion_love', 'tgt_type': 'glob', 'user': 'sudo_love'}

Event  chain  `test.ping`Event fired at Wed May 27 22:00:14 2015 ************************* Tag: salt/job/20150527220014773556/ret/mon1 Data: {'_stamp': '2015-05-27T20:00:14.981640', 'cmd': '_return', 'fun': 'test.ping', 'fun_args': [], 'id': ‘minion_love', 'jid': '20150527220014773556', 'retcode': 0, 'return': True, 'success': True}

Reactor

Ability  to  execute  commands  Allow  events  to  trigger  ac@ons

# This script is to be called from the reactor system {%- set application = data['data']['application'] %} {%- set container = data['data']['container'] %} {%- set node = data['data']['salt-node'] %} {%- set tag = data['data']['tag'] %}

sync-states: cmd.saltutil.sync_states: - tgt: {{ node }} deploy-containers: cmd.state.sls: - tgt: {{ node }} - arg: - containers.{{ container }} - kwarg: pillar: containers: imagetag: {{ application }}: {{ tag }}

Salt  ApiYes!  There’s  an  rest-­‐api

curl -sS salt.jacksoncage.se/run \ -H 'Accept: application/x-yaml' \ -d client='local' \ -d tgt='*' \ -d fun='test.ping' \ -d username='saltdev' \ -d password='saltdev' \ -d eauth='pam'

Salt  Api

# Example start creation dev box via curl: curl -H 'X-Salt-Key: Secret-Key-Here' \ -H 'Content-Type: application/json' \ -d '{"username":"love"}' \ https://salt.jacksoncage.se/hook/api/devbox/create

OrchestrateConfigura@on  order  Sequenced  applied  states  Orchestrate  deployment  in  order  on  mul@ple  systems

# Create AWS EC2 instance create-instance: salt.state: - tgt: {{ master }} - sls: devlovin.cloud.create - concurrent: True - pillar: devlovinreact: instancename: {{ basename }}{{ username }}

# Run highstate highstate-run: salt.state: - tgt: {{ basename }}{{ username }} - highstate: True - pillar: devlovinreact: username: {{ username }} - require: - salt: create-instance

BeaconsHook  into  system  processes    Translate  external  events  into  the  salt  event  bus  Events  can  then  be  responded  by  Reactor

Beacons

DEMO

Ques@ons?

Love  Nyberg  @jacksoncage  

love@bloglovin.com

Thanks!

Love  Nyberg  @jacksoncage  

love@bloglovin.com