How we-use-ansible

How we useAnsible Meetup - Lyon - 18 / 02 / 2016

@bdu_p

● root DevOops, automation @

● Distributed systems & Data

● Go - Python - Cobol

● Machine Learning

Who’s this guy ?

m6web.kafkam6web.php

m6web.ntpdm6web.spark

[...]

playbooks - ops

ansible-vault

control-masterServers

ElasticSearch / Kibana

Global Architecture

Roles

«This is the Unix philosophy: Write programs that do one thing and do it well.

Write programs to work together […]»Doug McIlroy

Roles

● Ex: ELK should *NOT* be a single role○ java○ elasticsearch○ logstash○ kibana○ nginx○ open-ssl○ ssl-certificates

Roles

● One role = One public repository

● ~ 60 repos

● Code review / Pull Requests

● Continuous integration

Is this your DevOps Larry ?

- name: Install python-software-properties apt: name=python-software-properties state=installed update_cache=yes cache_valid_time=86400

Roles - Code conventions and readability

- name: Install dependencies

apt:

name: “ {{ item }} “

state: installed

update_cache: yes

cache_valid_time: “{{ global_cache_valid_time }}”

with_items: “ {{ kafka_dependencies }}”



[ANSIBLE0004] Git checkouts must contain explicit versionexample.yml:36Task/Handler: using git module

[ANSIBLE0003] mismatched bracesexample.yml:2debug oops a missing {{bracket}

[ANSIBLE0002] Trailing whitespaceexample.yml:19 action: do nothing


λ ~ role-skeleton (master|✔) ls -1defaultsfileshandlersmetataskstemplatestestsvarsREADME.md


Keep main.yml as en entrypoint

Roles - Code conventions and readability---# tasks file for m6web.kafka

- name: Check if broker_id is set fail: msg: "You must set kafka_broker_id variable !" when: kafka_broker_id is not defined

- include: users.yml tags: - 'role::m6web.kafka' - 'role::m6web.kafka::users'

- include: packages.yml tags: - 'role::m6web.kafka' - 'role::m6web.kafka::packages'

- include: config.yml tags: - 'role::m6web.kafka' - 'role::m6web.kafka::config'

- include: service.yml tags: - 'role::m6web.kafka' - 'role::m6web.kafka::service'


Roles - Tags

Tag everythingminimum is role name

Roles - Tags

// Change ntpd config on *all* servers

$ ansible-playbook -i inventory playbooks/all.yml \

--tags ‘role::m6web.ntpd::config

// Change kafka config for dev

$ ansible-playbook -i inventory playbooks/dc1.yml \

--tags ‘role::m6web.kafka::config’ \

--limit lyon-dev

Roles - Versioning

None

Roles - Tests

« Testing leads to failure, and failure leads to understanding. »

Burt Rutan

Roles - Tests

pre-commit (by Yelp)

Ansible-lint

YAML syntax

Python jinja2 syntax

Playbook syntax

Idempotence

ServerSpec

One shell script run in

Roles - Tests - ServerSpec

describe package('cassandra') do it { should be_installed }end

describe service('cassandra') do it { should be_running }end

describe command('nodetool status') do its(:stdout) { should contain('UN 127.0.0.1') }end

Roles - Tests - ServerSpec

ports = [7199, 9160, 7000, 9042, 61621]

ports.each do |p| describe host('127.0.0.1') do it { should be_reachable.with( :port => p, :proto => 'tcp' ) } endend

Roles - Tests

Merge Master

Update Control-Master

git push

TestsUpdate P.R status

Playbooks

● Real orchestration

● Glue between roles, inventories, variables, and secrets

● Ops Playbook

● Packer / Vagrant Playbook

● Soon to come Docker Playbooks

Playbooks

● Groups of hosts (clusters, logical roles)

● Single Server (aptly, jenkins-master etc…)

● One shot (upgrade bash, upgrade openssl)

● Utilities (install_sysdig, upgrade_system)

Playbooks

Secrets

● Ansible SSH private key

● SSL private key

● Nginx passwords

● Unix passwords

● AWS / GCE Access keys

Secrets - What ?

● Private git repository

● ansible-vault encrypt - ansible-vault decrypt

● pre-commit hook prevent clear text to be commited

● Control Master has vault unsealed (manually)

Secrets - How ?

Scripting Ansible

$ ./get-package-version

--inventory ../inventory

--hosts all

--package "zsh" | jq "."

{

"down": [],

"detailled": {

"5.0.5-4": ["srv-x1"],

"5.0.2-3": ["srv-z1","srv-z2"],

"4.3.10-5": ["srv-y1","srv-y2"]

},

"summary": {

"5.0.5-4": 1,

"5.0.2-3": 2,

"4.3.10-5": 2

}

}

Scripting Ansible (not 2.0)


import ansible.runner

from collections import defaultdict

import json

import argparse

Scripting Ansible (not 2.0)runner = ansible.runner.Runner(

module_name='shell',

module_args=' '.join(["dpkg-query",

"-f=\'${Version}\' -W",

args['package']]),

pattern=args['hosts'],

remote_user=args['user'],

inventory=inventory,

forks=10

)

res = runner.run()


for host in sorted(res['contacted'].keys()):

version = res['contacted'][host]['stdout']

cnt['summary'][version] += 1

cnt['detailled'][version].append(host)

print json.dumps(cnt)

$ ./get-package-version

--inventory ../inventory

--hosts all

--package "zsh" | jq "."

{

"down": [],

"detailled": {

"5.0.5-4": ["srv-x1"],

"5.0.2-3": ["srv-z1","srv-z2"],

"4.3.10-5": ["srv-y1","srv-y2"]

},

"summary": {

"5.0.5-4": 1,

"5.0.2-3": 2,

"4.3.10-5": 2

}

}



● How many servers with kernel 3.13.0-43 ?

● How many VMs > 12G ram ? with 2 vCPUS ?

● Which server needs reboot ?

● Which server needs security updates ?

● List all servers with uptime less than X minutes

● Index all ohai facts in Elasticsearch

● Every night, run a --check --diff

○ Did we silently changed something today ?


+ + ++

● Simple but powerful orchestrator over SSH

● Agent / Server Less

● Dynamic inventories

● Unix Philosophy

● Python <3 <3 <3

● Community

What we <3

● No curated roles or playbook

○ simple != manageable

○ steep learning curve (best practices + team work)

● No emphasis on tests (roles / playbook)

● Conditionnals can be tricky

● Ansible-vault

What we <3 (less)

● v2 (core + API)

● Goss: ServerSpec in Go

● Pew or conda VS virtualenv

● Ansible runner + Celery = <3 (some caveats)

● HashiCorp Vault

Dig !

Questions ?

@bdu_p#lolcat #devoops

tech.m6web.freffetcapital.tumblr.com

How we-use-ansible

Technology

Transcript of How we-use-ansible