Deploying Complex Stacks with AnsibleDevconf 2020 - 2020-01-24

Will Foster • @sadsfae github.com/sadsfae • https://hobo.house

● Client-less configuration management system● Written in Python● Uses SSH as a transport mechanism● Uses YAML for logic and tasks● Uses Jinja2 for templating

What is Ansible?

● Save time and resources so you can do other things● Significantly lower deployment time for apps / services● Reduce complexity and human error via automation

Why should I use Ansible?

● Strive for idempotency. ○ Ansible should do nothing if in desired state

● Template as many configuration files as possible● Break deployment pieces/objectives into logical parts● Make liberal use of configuration variables● Aim for an open-ended design and choice● Use Ansible provided modules wherever possible

Configuration Management Goals

How Baby Yoda Writes Ansible

● Deploy a full all-in-one ELK/EFK 6.8.x stack○ Elasticsearch (search engine, time-series datastore)○ Logstash (data collection, log parsing engine)

○ Kibana (analytics, visualization)

○ Nginx (web reverse proxy)

● We’ll use CentOS7

● Code here: github.com/sadsfae/ansible-elk

Complex Stack Example: ELK/EFK

● host-02 (client) →○ Send system logs via Filebeat to Logstash on host-01

● host-01 (server) → ○ Logstash accepts system logs over SSL/TLS○ Logstash filters logs and sends to Elasticsearch○ We visualize events in Kibana Web UI

DEMO: ELK/EFK Deployed via Ansible

You’ve had your snack, you’ve played with the buttonNow it’s time to put your jammies on.

Ansible Facts can Customize your Environment

● Example: automatically tune Elasticsearch JVM heapsize based on amount of physical memory detected

Use System Facts to Adjust Configuration

Handling Service Dependencies in Ansible

● When components depend on other components, check they are available before proceeding

● Can be done by HTTP return code, port, or web content

Make use of Service Dependency Checking

● wait_for, until and uri Ansible modules are useful for this

Make use of Service Dependency Checking

● Parent service availability checks during a playbook run

Make use of Service Dependency Checking

● Another example: checking raw output matches what we want before proceeding

Dependency Checking Example - Dell Racadm

Using Variables for Conditional Logic

● Complex stacks will inevitably grow to require more deploy options● Maximize the usage of conditional vars to provide choice● Expand options for deployment flexibility (see all.yml)

../install/group_vars/all.yml → → → →

vars_files Make your Playbook more Flexible

Organizing your Playbook and role Hierarchy

Common Design Hierarchy for Large Playbooks

Try to use one role per major component

● Use branches to support older versions/series of the stack● Use branches to support deployment to different environments● Gitlab.com offers free, private repositories each with 10Gb of space

Keep your Playbooks in an SCM (Git, etc).

Automate Client Operations When Possible

● Make it easy to automate client integration● e.g. SSL/TLS certificate retrieval, client applications / libraries

Make sure you’re automating client-side

CI / CD and Ansible Lint

● Use ansible-lint to test your playbooks

ansible-lint install/*.yml -v

ansible-lint install/roles/*/*/*.yml

Use Travis CI to run Lint on Repos

Full guide at https://hobo.house

Troubleshooting and Debugging Tips

● Useful to determine registers, variable output and stdout

Using Debug in a Playbook

● Debug can be useful for informational messages

Using Debug in a Playbook

Running ansible-playbook --check tells you what it would do

● Upgrade ELK Stack to 7.x+ (currently at 6.8.x)

● Support multi-node deployments

ansible-elk roadmap

We use ELK Stack for recording QUADS data

● Manages bare-metal systems and network switch automation based on schedules set in the future

● Powers automation within the Red Hat Scale Lab

● https://quads.dev● github.com/redhat-performance/quads

QUADS is an automation framework

● github.com/sadsfae

● https://hobo.house

● Twitter: @sadsfae

● Freenode IRC: sadsfae

Thank you for attending!

Questions, Comments, Discussion?