Post on 28-May-2020
digitalocean.com
Introduction
digitalocean.com
What does DO do?
Simple, Developer-focused Cloud Hosting
digitalocean.com
What are we using
Ansible for?
digitalocean.com
Example Deployment
digitalocean.com
Example Project Layout
digitalocean.com
Project Layout
● Inventories
● Local Module Library
● Group Variables / Host Variables
● Roles
○ Component roles
○ Project Specific Roles
● Playbooks
○ Server Templates
○ Cluster Configuration
○ Actions
● Makefiles
digitalocean.com
Inventories
● List hosts (by environment)
● Define groups
● Guardrails
ansible-playbook -i inventories/development ...
digitalocean.com
Inventories
● List hosts (by environment)
● Define groups
● Guardrails
all:
children:
mysql:
children:
mysql_managed:
hosts:
test-mysql-0[1:3].atlantic.com:
test-mysql-0[1:6].pacific.com:
mysql_unmanaged:
digitalocean.com
Inventories
● List hosts (by environment)
● Define groups
● Guardrails
ansible-playbook ... --extra-vars="target_env=development” ...
Playbook:
---
- hosts: mysql:!mysql_unmanaged:&{{ target_env }}
...
digitalocean.com
Inventories: Constructed Groups
plugin: constructed
strict: false
groups:
dev: inventory_hostname.startswith('dev-')
digitalocean.com
Inventories: Constructed Groups
plugin: constructed
strict: false
groups:
dev_mysql: (group_names|intersect(['mysql', 'dev']))|length >= 2
digitalocean.com
Inventories: Ordering
inventories
- development
- 10_mysql.yml
- 90_environment.yml
- 99_dev_mysql.yml
- production
- staging
digitalocean.com
Variable Order of Precedence
1. command values (eg “-u user”)2. role defaults3. inventory file or script group vars4. inventory group_vars/all5. playbook group_vars/all6. inventory group_vars/*7. playbook group_vars/*8. inventory file or script host vars9. inventory host_vars/*
10. playbook host_vars/*11. host facts / cached set_facts
12. play vars13. play vars_prompt14. play vars_files15. role vars (defined in role/vars/main.yml)16. block vars (only for tasks in block)17. task vars (only for the task)18. include_vars19. set_facts / registered vars
20. role (and include_role) params21. include params22. extra vars (always win precedence)
digitalocean.com
Variable Management
● Role defaults interface with the role● Define project level generic variables applicable to all environments
○ playbook group_vars/all○ playbook group_vars/*
● Host specific overrides○ inventory host_vars/*
● Variables we construct○ role vars / include_vars / set_facts
● Functional role variables○ role (and include_role) params
● Guardrails○ extra vars (always win precedence)
digitalocean.com
Variable Management Example - Defaults---### proxysql installproxysql_create_image: "{{ global_create_image | default(false) }}"proxysql_download_src: https://github.com/sysown/proxysql/releases/downloadproxysql_version: 1.4.10proxysql_mysql_client_version: 5.7
proxysql_user: proxysqlproxysql_group: proxysqlproxysql_datadir: /var/lib/proxysqlproxysql_restart_missing_heartbeats: 10
...
# autocommitproxysql_mysql_autocommit_false_is_transaction: falseproxysql_mysql_autocommit_false_not_reusable: falseproxysql_mysql_enforce_autocommit_on_reads: falseproxysql_mysql_forward_autocommit: false
...
digitalocean.com
Variable Management Example - Vars---...
### percona required packagesproxysql_release: "{{ proxysql_download_src }}/v{{ proxysql_version }}/proxysql_{{ proxysql_version }}-ubuntu18_amd64.deb"
...
proxysql_mysql_variables: autocommit_false_is_transaction: variable: "autocommit_false_is_transaction" variable_value: "{{ proxysql_mysql_autocommit_false_is_transaction | to_json }}" autocommit_false_not_reusable: variable: "autocommit_false_not_reusable" variable_value: "{{ proxysql_mysql_autocommit_false_not_reusable | to_json }}" client_found_rows: variable: "client_found_rows" variable_value: "{{ proxysql_mysql_client_found_rows | to_json }}"
...
digitalocean.com
Variable Management Example - Config#jinja2: lstrip_blocks: "true"datadir="{{ proxysql_datadir }}"restart_on_missing_heartbeats={{ proxysql_restart_missing_heartbeats }}
admin_variables={{% for config_item in proxysql_admin_variables|dictsort %} {% if config_item.1.variable_value is not none %}
{{ config_item.1.variable }}={{ config_item.1.variable_value | to_json }}{% endif %}
{% endfor %}}
mysql_variables={{% for config_item in proxysql_mysql_variables|dictsort %} {% if config_item.1.variable_value is not none %}
{{ config_item.1.variable }}={{ config_item.1.variable_value | to_json }}{% endif %}
{% endfor %}}
digitalocean.com
Anatomy of a Role
digitalocean.com
Anatomy of a Role● A role should be map to a single unit of functionality that utilise a common set of variables.
● Roles should be intuitive, and wherever possible mimic a common structure.
● Role Variable Management
○ Where possible, a [component] role should be generic, and any variables should map
to sensible defaults.
○ The interface into role customisation should be via scalar role defaults.
○ Role variables should be used for variables that shouldn't be overridden in normal
circumstance, or as syntactic sugar to construct variables internal to the role.
● A role should have repeatable logic and should avoid logical branching that might be
non-repeatable.
digitalocean.com
Component Roles
digitalocean.com
Role Versioning
- name: role_mysql_proxysql
src: git+ssh://git@github.pacific.com/ansible/role_mysql_proxysql.git
version: 1.1.1
digitalocean.com
Example ProxySQL Deployment
digitalocean.com
Testing Roles
digitalocean.com
● pip install --user molecule
○ pip install --user molecule[ec2]
○ pip install --user molecule[docker]
Molecule
digitalocean.com
● create / destroy / list / cleanup
● prepare
● dependency
● login
Molecule Commands
digitalocean.com
Anatomy of a Role
digitalocean.com
● lint
● syntax
● idempotence
● verify
● check
Molecule Commands
digitalocean.com
● converge
● test
● side-effects
Molecule Commands
digitalocean.com
Role Development with Molecule
digitalocean.com
Testing ProxySQL Example
digitalocean.com
Molecule Configuration
dependency: name: galaxydriver: name: dockerlint: name: yamllintplatforms: - name: host1 image: "geerlingguy/docker-${MOLECULE_DISTRO:-ubuntu1804}-ansible:latest" command: ${MOLECULE_DOCKER_COMMAND:-""} volumes: - /sys/fs/cgroup:/sys/fs/cgroup:ro privileged: true pre_build_image: trueprovisioner: name: ansible lint: name: ansible-lint
digitalocean.com
Molecule Configuration
scenario: name: default converge_sequence: # - dependency - create # - prepare - converge test_sequence: - lint - destroy # - dependency - syntax - create # - prepare - converge - idempotence # - side_effect - verify - destroy
digitalocean.com
Molecule Configuration
verifier: name: testinfra env: PYTHONWARNINGS: "ignore:.*U.*mode is deprecated:DeprecationWarning" options: v: 1 lint: name: flake8
digitalocean.com
Functional Testing with TestInfraproxysql_file_attributes = ("proxysql_file," "proxysql_file_user," "proxysql_file_group," "proxysql_file_mode")
@pytest.mark.parametrize(proxysql_file_attributes, [ ("/root/.my.cnf", None, None, 0o600), ("/etc/proxysql.cnf", "proxysql", "proxysql", 0o644),])def test_proxysql_files(host, proxysql_file, proxysql_file_user, proxysql_file_group, proxysql_file_mode): f = host.file(proxysql_file)
assert f.exists assert f.is_file if proxysql_file_user: assert f.user == proxysql_file_user if proxysql_file_group: assert f.group == proxysql_file_group if proxysql_file_mode: assert f.mode == proxysql_file_mode
digitalocean.com
Functional Testing with TestInfra
● Host fixture
○ host.file
○ host.package
○ host.service
○ host.run
digitalocean.com
Continuous Integration Pipeline
digitalocean.com
User Management
digitalocean.com
● Deployment and maintenance of individual and Service users
● Maintain user (dynamic) privileges
● Manage secrets (securely)
● Manage across both MySQL and ProxySQL
User Management Story
digitalocean.com
Manual Worst Case Scenario
sammy
sammy@10.21.%
sammy@10.22.%
sammy@10.23.%
sammy@10.24.%
sammy@10.25.%
digitalocean.com
User Management Requirements
● Deploy user control manifest to Ansible Project Role
● Generate & Encrypt secrets in Ansible Vault
● Consistent Delivery across technologies / tenancies / environments
● Scalable solution
digitalocean.com
Request > New UMC > Gen Secret > commit/PR > Peer Review > Dry Run > Deploy
New User Deploy Chain
+ >> >> >>
digitalocean.com
User Control Manifestsammy_ro: state: present active: true default_shard: atlantic default_hostgroup: swimming_pool default_schema: baby_shark enabled_schemas: - baby_shark - left_shark cluster_privs: atlantic: env: dev: - '%’ privs: - 'baby_shark.*:SELECT' pacific: env: dev: - '%’ privs: - 'left_shark.*:SELECT,INSERT,UPDATE,DELETE'
digitalocean.com
docs.ansible.com/ansible/latest/modules/list_of_database_modules.html?#proxysql
proxysql_mysql_users
proxysql_query_rules
proxysql_replication_hostgroups
proxysql_scheduler
ProxySQL Ansible Module
proxysql_backend_servers
proxysql_global_variables
proxysql_manage_config
digitalocean.com
Secret Storage
digitalocean.com
Secret storage
Ansible Vault
● Used for shared secrets● Fine-grained access control
● Simple, no dependencies● Stored in repo w/ inventory &
playbooks
digitalocean.com
Ansible Vault
Vault file per environment (dev/stage/production)
---monitoring_password: ooxalohquaiK8aidai0xbackup_password: ahvowooG9dohfashaCho
ansible-vault encrypt
$ANSIBLE_VAULT;1.2;AES256;production3538653766323031333335306239343730333830653630373731383863373638616461633438363135666136383566336661383263383566376361353637636231623034380a3263383163623935383666633463656664653532666338656539633430636266656230626232316662386639343233656439
digitalocean.com
Ansible Vault
sammy_ro: dev: '*100569F51F55F3599CECBCABB9AC59AB29F30283' stage2: '*712E505DB01097F24D1B72509A820E254A525528' production: '*1A0174A0F2F4692917994C19C158F96B1914A53F'sammy_rw: dev: '*0A5B11132F429A38988540A351C60A620DE5BBFA' stage2: '*8B5CCFF05B945DED591D603DF936805476E77837' production: '*C0BF0057A4AC079E7ABE2FEC54DC87A1680B95DC'
Two vault files for all password hashes
● Passwords distributed to users via LastPass● Ansible never needs password, just hash
digitalocean.com
Ansible Vault
Challenges of ansible-vault approach:
● Merge conflicts● Visibility / discoverability
digitalocean.com
diffing Ansible vaults
$ git diff individuals.yml $ANSIBLE_VAULT;1.1;AES256-6530653733663966316630613666393-6139303463613133633262643034633-3235346162366237366538653361653...+3938303335363632343730656164356+3433373263346136653730313636336+3862633932353039616334303631376
digitalocean.com
diffing Ansible vaults
.gitconfig:[diff "ansible-vault"]
textconv = ansible-vault viewcachetextconv = false
.gitattributes:individuals.yml diff=ansible-vault
digitalocean.com
diffing Ansible vaults
$ git diff individuals.yml sammy_rw: dev: '*0A5B11132F429A3898854' stage2: '*8B5CCFF05B945DED59837'- production: '*C0BF0057A4AC0780B95DC'+ production: '*B1E8174F9DCE654C25608'
digitalocean.com
Similar config for merge
/usr/local/bin/ansible-vault-merge
.gitconfig:[merge "ansible-vault"]
name = ansible-vault merge driverdriver = /usr/local/bin/ansible-vault-merge -- %O %A %B %P
.gitattributes:individuals.yml diff=ansible-vault merge=ansible-vault
See github.com/building5/ansible-vault-tools
digitalocean.com
Finding variables in vault
Where is backup_password defined?
● Variable prefix○ In vars.yml: backup_password: "{{ vault_backup_password }}"○ In vault.yml: vault_backup_password: ahvowooG9dohfashaCho
● git grep:$ git grep --textconv backup_passwordvars.yml:backup_password: "{{ vault_backup_password }}"vault.yml:vault_backup_password: ahvowooG9dohfashaCho
digitalocean.com
Ansible vault passwords
● Store in local file○ Exclude from git!○ Can be executable
● Can define multiple vault-ids:
[defaults]vault_identity_list = production@vault-keyring-client.py,stage@vault-keyring-client.py
digitalocean.com
HashiCorp Vault
For secrets shared with others
digitalocean.com
HashiCorp Vault
# Set VAULT_ADDR$ vault login# Set VAULT_TOKEN
$ vault write secret/sammy pass=Za6Uoy
$ vault read secret/sammyKey Value--- -----pass Za6Uoy
digitalocean.com
HashiCorp Vault
● hashi_vault lookup plugin○ https://docs.ansible.com/ansible/latest/plugins/lookup/hashi
_vault.html
● Various auth methods○ We use approle and token
● Auth secret stored in environment variable or in Ansible vault
digitalocean.com
HashiCorp Vault
- copy: content: >- {{ lookup('hashi_vault', 'secret=secret/sammy_cert:cert url=https://vault.example.com:8200 auth_method=approle role_id={{ role_id }} secret_id={{ secret_id }} ) }} dest: /etc/nginx/ssl/certificate.crt
digitalocean.com
HashiCorp Vault
Why not HashiCorp Vault for everything?
● External dependency● Changing policies requires another commit and
approval by another team
digitalocean.com
Ansible Performance
digitalocean.com
Performance Visibility
ansible.cfg
callback_whitelist = profile_tasks, profile_roles, timer
stdout_callback = actionable
digitalocean.com
Performance
● SSH optimisations
● Gathering Facts
● Play Logic
digitalocean.com
Performance
● SSH optimisations
● Gathering Facts
● Play Logic
digitalocean.com
A mitogen is a chemical substance, usually a protein, that induces a cell to begin cell division
Mitogen for AnsibleAn Ansible Plugin that replaces the use of SSH
digitalocean.com
Mitogen for Ansible
● Redesigned UNIX connection layer and module runtime for Ansible
● Easy installation, minimal configuration
● Immediate gains (immediately) 1% - 7% improvement in run duration
*unless you have and it didn’t help (Windows, Networking equipment)
digitalocean.com
Performance Challenges
● Loopy Loops
digitalocean.com
Module Override
digitalocean.com
With pushdown Without mitogen
digitalocean.com
With pushdown With mitogen
digitalocean.com
Without pushdown Without mitogen
digitalocean.com
Without pushdown With mitogen
digitalocean.com
Mitogen SSH Only
Standard mysql_user 4m 23s 10m 6s
Patched mysql_user 7m 33s 8m 19s
We’re Hiring!
https://www.digitalocean.com/careers