Declarative Infrastructure Tools

DeclarativeInfrastructureTools

Alex ConwayRelease Automation Engineer,

[email protected]

Who Am I?● Former particle physics guy● Release Automation Engineer● Boston● AWS, Linux, Python, Ansible, Docker

DISCLAIMER:

What’s This All About?● Some tools I like.● Some fun projects I’ve worked on.● Declarative Tools.● Creating simple interfaces to powerful

tools.● Maybe change the way you think about

your next project.

What’s It All For?● Need to create many independent test

environments.● Different sizes and configurations.● Give devs power to make their own.

About DataRobot

https://www.datarobot.com/



The Company Our Vision:● To become the world’s go-to resource for data science

and machine learning tools

Our Product:● Automates the processes of data science● Creates highly accurate predictive models● Runs on open-source machine learning libraries● Allows data scientists to work faster and smarter● Both cloud and enterprise/on-prem

DataRobot Application Stack● Application

○ Python○ Node.js

● Database○ MongoDB○ Redis

● Storage○ S3/Gluster/HDFS

Docker!

ImperativevsDeclarative

Imperative vs DeclarativeDeclarative● Please give me a

donut

Imperative● Please make some

dough● Then shape it into a

circle● Then fry it

Imperative vs DeclarativeDeclarative● Describe a state● Explicit

dependencies● Simple● Hard

Imperative● Describe a process● Implicit

dependencies● Complex● Easy

Makeby GNU

https://www.gnu.org/software/make/



WTF is… Make?● Build tool with

declarative syntax● Dependency graph● Idempotent● Do any step out of order# File: Makefile

raw.dough:

./make_dough.sh

circle.dough: raw.dough

./make_circle.sh raw.dough

donut: circle.dough

./fry.sh circle.dough

$ make donut

./make_dough.sh

./make_circle.sh raw.dough


make: execvp: ./fry.sh: Permission denied

make: *** [donut] Error 127

$ sudo make donut


Okay.

_.-------._ .' ___ '. / (___) \ |'._ _.'| | `'-----'` | \ / '-.______..-'

Terraformby Hashicorp

https://www.terraform.io/



WTF is… Terraform?

● Infrastructure as Code○ Describe your whole infrastructure in simple, declarative

configuration language.○ AWS, DigitalOcean, VSphere, OpenStack, and more.

● Resource Graph○ Track dependencies between resources.

● Mutate State○ Transition between states with ease, updating all affected

resources.

“Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently.” - https://www.terraform.io/intro/index.html

https://www.terraform.io/intro/index.html

# File: test/terraform-test.tf

provider "aws" {

region = "us-east-1"

}

resource "aws_instance" "hello_world" {

ami = "ami-deadbeef"

availability_zone = "us-east-1a"

instance_type = "t2.micro"

key_name = "my_key"

subnet_id = "subnet-deadbeef"

tags = {

name = "tf-test"

}

vpc_security_group_ids = ["sg-deadbeef"]

}

Basic Usage With AWS$ cd test

$ ls

terraform-test.tf

$ terraform plan

...

Plan: 1 to add, 0 to change, 0 to destroy.

$ terraform apply

...

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

$

● Store Terraform state in shared location, not your hard drive.

● S3, Hashicorp Atlas, any REST service.

● Split code between folders or repositories but share common state.

● Currently it does not protect from simultaneous changes to state!

Remote State$ terraform remote config \

-backend=s3 \

-backend-config="region=us-east-1" \

-backend-config="bucket=terraform-state" \

-backend-config="key=test.tfstate"

$ terraform remote pull

$ terraform plan

$ terraform apply

$ terraform remote push

$

Makefile for TerraformSimple command wrapper to standardize Terraform workflow.# File: Makefile

...

remote-config:

terraform remote config $(remote_vars)

remote-pull: remote-config

terraform remote pull

get:

terraform get

plan: get remote-pull

terraform plan $(tf_vars)

apply: plan get remote-pull

terraform apply $(tf_vars); \

terraform remote push

Terraform● Describe state● Migrate state● Share state● Infrastructure as

Code!

● Write out instructions○ Declarative tasks,

imperative plays● Hard to keep track

of state● Hard to make

arbitrary changes

Ansible/scripts

DockerComposeby Docker

https://docs.docker.com/compose/



WTF is… Docker Compose?

● YAML config file for defining Dockerized services○ Image, command, ports, volumes, links, etc.○ Declarative!

“Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a Compose file to configure your application’s services. Then, using a single command, you create and start all the services from your configuration.”- https://docs.docker.com/compose/overview/

# File: docker-compose.yml

# Adapted from https://docs.docker.

com/compose/wordpress/

---

wordpress:

build: ./wordpress/

command: php -S 0.0.0.0:8000 -t /code

ports:

- "8000:8000"

links:

- mysql

volumes:

- wordpress/:/code

mysql:

image: orchardup/mysql

environment:

MYSQL_DATABASE: wordpress

$ docker-compose up -d

Creating mysql_1...

Building wordpress...

...

Successfully built efe76b2be23f

Creating wordpress_1...

$ curl localhost:8000

Welcome to Wordpress!

$

Docker Compose Example

infrastructure.tfdocker-compose.yml

Dev Ops

???

Common Solutions● Config Management

○ Chef, Ansible, Puppet● Networking

○ libnetwork, libkv● Service discovery

○ Consul, Zookeeper, Etcd● Scheduling

○ Mesos, Kubernetes

Common Problems● Dev != Prod● Prod is very complex

○ Devs don’t know it● Infrastructure tied to services● Services tied to infrastructure

infrastructure.tfinfrastructure.tf.json

docker-compose.yml layout.yml

ltparse

container-from-compose

DevOps!

ltparseby DataRobot

https://github.com/datarobot/ltparse



ltparse: ‘Flexible’ Terraform● Want many variations● Writing Terraform can be repetitive● Hard to read and write● Not designed with flexibility in mind● Want more simple cluster definitions.● Want to integrate with configuration

management.

ltparse: ‘Flexible’ Terraform● Input YAML file:

○ Servers○ Security groups○ Elastic Load Balancers○ route53 DNS records

● Python○ Defaults○ Update with YAML config○ Write parsed.tf.json file

● terraform apply

ltparse Example# File: flexible/layout.yaml

---

servers:

- label: webserver

services:

- wordpress

- nginx

route53_record: webserver

instance_info:

instance_type: m4.xlarge

- label: db

services:

- mysql

route53_records:

- label: webserver

public: True

$ cd ../flexible

$ ls

layout.yaml

$ ltparse layout.yaml

$ ls

layout.yaml parsed.tf.json

$ export cluster_id=test-cluster

$ make plan

...

Plan: 4 to add, 0 to change, 0 to destroy.

$

ltparse Output "resource": {

"aws_instance": {

"db": {

"ami": "${var.ami_ids.hvm}",

"associate_public_ip_address": true,

"availability_zone": "us-east-1a",

"count": 1,

"instance_type": "m4.large",

"key_name": "${var.aws.key_name}",

"subnet_id": "${terraform_remote_state.subnets.output.us-east-1a}",

"tags": {

"id": "db_${var.run_id}",

"label": "db",

"owner": "test_${var.run_id}",

"user": "${var.build_user}"

},

"vpc_security_group_ids": [

"${module.default_security_group.id}"

]

... etc

ltparse● trafaret schema for layouts

○ Good, fast feedback● click for cli

○ No boilerplate○ Typed parameters

● py.test for testing○ Fixtures○ Test good/bad layouts

● setuptools for packaging

ltparse Bonuspy.test fixtures for testing layout parser

# File: tests/layouts/bad/no_target.yaml

---

route53_records:

- label: bad

domain: domain

servers:

- label: server

expects: !!python/object/apply:ltparse.

parser.ConfigurationError [route53 record

label `bad` not applied to any instances or

elbs]

def test_full_layouts_bad(test_bad_layout): """ For each layout in tests/layouts/bad, assert that running

format_data fails with the exception and message defined in

the layout. """ expected_exception = test_bad_layout['expects'] with pytest.raises(type(expected_exception)) as excinfo: format_data(test_bad_layout) assert expected_exception.message == str(excinfo.value)

infrastructure.tf.json


ltparse


ContainerFromComposeby DataRobot

● An Ansible Role○ Take compose file○ Take layout○ Take infrastructure○ Make distributed app

● Benefits○ One config for dev and prod○ Simple input, simple output○ Generic deployment

WTF is… Container From Compose?

Compose and Layout# File: layout.yaml

---

servers:

- label: web

services:

- wordpress

route53_record: wordpress

instance_info:

instance_type: m4.xlarge

- label: db

services:

- mysql

route53_records:

- label: wordpress

public: true

# File: docker-compose.yml

---

wordpress:

build: ./wordpress

command: php -S 0.0.0.0:8000 -t /code

links:

- mysql

mysql:

image: orchardup/mysql

ports:

- "3306:3306"

environment:

MYSQL_DATABASE: wordpress

MYSQL_USER: wordpress

MYSQL_PASSWORD: pass

Ansible Playbook● Traditionally:

○ Very imperative○ Do this, do that, then you

have a site○ Application-specific○ Config is distributed; hard to

understand final state● Now:

○ Very declarative○ Config is centralized○ Very generic

# File: inventory/site.inventory

[mysql:children]

tag_id_db_talk_test

[wordpress:children]

tag_id_web_talk_test

# File: site.yml

---

- hosts: mysql:wordpress

vars_files:

- docker-compose.yml

- services.yml

roles:

- container-from-compose

Container-From-Compose

● Install dependencies○ Use role meta to create

dependency graph!● Copy code/config● Build any images● Start containers

1 ---

2 - name: Example services list

3 set_fact:

4 services_list: "{{group_names | intersect(defined_services)}}"

5

6 - name: Start docker containers (simplified)

7 docker:

8 name: "{{item}}"

9 image: "{{compose_vars[item]['image']}}"

10 command: "{{compose_vars[item]['command'] | default(omit)}}"

11 env: "{{drenv}}"

12 volumes: "{{compose_vars[item]['volumes'] | default(omit)}}"

13 with_items: services_list

container-from-compose example

Caveats● Very dense Ansible code● Some rough edges and limitations

○ Links○ --net=host, /etc/hosts networking

● Doesn’t currently handle state changes○ eg. can’t move service between hosts.

infrastructure.tf.json


ltparse


Thanks!

Bonus: why not Docker Swarm?● I’d love to try it● Active development

○ Swarm was experimental when we started○ Compose + Swarm is still experimental

● Swarm filters not yet supported○ Can’t loc service to node (like NGINX to instances

with ELB)

Declarative Infrastructure Tools

Engineering

Transcript of Declarative Infrastructure Tools