Container Orchestration

RoundupTrammell Scruggs,Arthur Berezin,

GigaSpaces

Agenda◇ Orchestration 101◇ Method of comparison ◇ Tools overview

Orchestration 101

Orchestration 101◇ Common Characteristics￭ Use DSL to define “blueprint”￭ Execute a process based on input from the blueprint ￭ Pass context information between the deployed

entities

◇ Different assumptions lead to different approaches ￭ Application Architecture￭ Infrastructure ￭ Scope of automation

Method of Comparison◇ Same application requirements ◇ “Production grade” deployment ◇ Broken into three main groups￭ Pure Play – Cloudify/TOSCA,

Terraform￭ Container Centric – Kubernetes￭ Infrastructure Centric - OpenStack Heat

◇ PaaS: CloudFoundry, OpenShift, DEIS◇ Config Management: Chef, Puppet, Ansible, Salt◇ Cloud Application Management: Apache Brooklyn, Juju◇ Cloud management platforms: RH CloudForms, Scalr◇ Cluster schedules: Mesos, Fleet, Yarn, Nomad

This is by no means an exhaustive comparison!

A Lot of Relevant Tools Which We Won’t Cover Today :(

The Test Application Test Application

◇ Manage dependencies◇ Reproducible ◇ Cloneable◇ Recoverable◇ Updateable

Common Requirements

◇ Setup monitoring and log collection ◇ Manual/Auto healing ◇ Manual/Auto scaling ◇ “Day 2”￭ Backup and restore ￭ Update code ￭ Infrastructure upgrades and patches

Beyond Just Setup

“Pure-Play” Orchestration

Terraform by Hashicorp

Introduction to Terraform◇ Part of the Hashicorp “ecosystem”◇ Command line tool ◇ Simple (in a good way) configuration◇ Extendable using plugins

Solution Overview

◇ Single top level configuration file ◇ Create all infrastructure install and

configure application with scripts.

Sample Configuration resource "openstack_compute_instance_v2" "mongod_host" { count = "3" region = "" name = "mongod_host" image_name = "${var.image_name}" flavor_name = "${var.flavor_name}" key_pair = "tf-keypair-1" security_groups = ["mongo_security_group"] network { uuid = "${openstack_networking_network_v2.tf_network.id}" } ... provisioner "remote-exec" { scripts = [ "scripts/install_mongo.sh" "start_mongod.sh" ] }}

Deployment Management

◇ Option 1: Static Allocation/Provisioning in the configuration file.

◇ Option 2: Third-party Tool￭ Consul/Surf￭ Other, e.g. AWS Autoscale

Cons: ◇ Configurations are not

portable across cloud providers

◇ Outsources advanced configuration management to other tools

◇ Context awareness is not built in

Terraform: Pros and Cons Pros: ◇ Simple & elegant◇ Solid Openstack

support◇ IaaS & Framework

neutral◇ Display Plan before

deployment◇ Idempotency◇ Hashicorp stack

TOSCA/Cloudify

TopologyNodes,

relationships, requirements &

capabilities

Workflows

Lifecycle operations

implementation (install, scale…)

PolicyEvents,

conditions, action

mapping

Topology Orchestration Spec for Cloud Applications

What is TOSCA?

What is TOSCA?◇ Standard for Cloud Orchestration◇ Describes how to map application

requirements to infrastructure capabilities

ApplicationBlueprint(TOSCA)

IaaSPlugins

ContainerPlugins

Conf MgmtPlugins

● Provision● Configure● Monitor● Manage

Monitoring &Alarming

Cloudify: an Open Source Implementation of TOSCA

Solution Overview◇ Single blueprint file with peripheral

scripts and plugins◇ Create all infrastructure install and

configure application with scripts◇ Describe future application

management and workflows◇ Relationships describe which code to

execute when one node event affects another node.

nodecellar_container: type: nodecellar.nodes.NodecellarContainer properties: Name: nodecellar image: Repository: nodecellar relationships: - type: node_contained_in_nodejs target: nodejs_host

Sample Configuration

nodejs_host: type: cloudify.openstack.nodes.Server interfaces: cloudify.interfaces.monitoring: start: implementation: diamond.diamond_agent.tasks.add_collectors inputs: collectors_config: default: CPUCollector: {} MemoryCollector: {} LoadAverageCollector: {} DiskUsageCollector: config: devices: x?vd[a-z]+[0-9]*$ NetworkCollector: {}

Sample Configuration

Deployment Management◇ Manager is aware of node instances’

state◇ logging and monitoring◇ Policy engine handles metric streams◇ Manager auto-executes workflows in

response to lifecycle requirements

◇ Built in workflows ￭ Install￭ Uninstall￭ Heal￭ Scale￭ Execute_operation￭ (Single common ‘cfy install’ in 3.4 )◇ Dependency awareness through graph

navigation◇ Remote/Local execution

Handling Post Deployment through Workflow & Policies

Cons: ◇ Complex◇ Delay in full support of

TOSCA spec◇ Not enough

implementations - user needs to write most of it

TOSCA/Cloudify: Pros and Cons

*Implementation specific

Pros: ◇ Infrastructure &

Framework neutrality ◇ Complete Life Cycle

Management ◇ Handles

Infrastructure & Software

◇ Post deployment handling*￭ Monitoring￭ Logging

◇ Can be tied to any peripheral system

Container OrchestrationKubernetes

Container deployment management ◇ Pods: a group of containers usually run together◇ Replication controller: ensures state of pod◇ Networking: pods are networked together◇ Service: Load balanced endpoint for a set of

pods

Quick Intro to K8s

K8s Architecture

Solution Overview◇ Single configuration file ◇ Uses pools of infrastructure resources◇ Describe future application

management and workflows◇ Relationships are maintained through

Pods

apiVersion: v1kind: Servicemetadata: name: nodecellar-servicespec: selector: app: nodecellar ports: - protocol: TCP port: 3000 targetPort: 3000 nodePort: 30000 type: NodePort

Sample ConfigurationapiVersion: v1kind: ReplicationControllermetadata: name: nodecellarspec: replicas: 2 selector: app: nodecellar template: metadata: name: nodecellar labels: app: nodecellar spec: containers: - name: nodecellar Image: nodecellar workingDir: / command: ["bash","start.sh"] ports: - containerPort: 3000 hostIP: 0.0.0.0

Cons: ◇ Not a lot of infrastructure

focus◇ Limited Openstack

support (mostly Rackspace)

◇ No topology awareness◇ Complex setup

Kubernetes: Pros and Cons Pros: ◇ (almost) zero

configuration autoheal

◇ Out of the box load balancer

◇ Simple scaling◇ Advanced behaviors

are pretty simple to implement

Infrastructure Centric

What Is Heat?

Provides a mechanism for orchestrating OpenStack resources through the use of modular templates

Heat Architecture

OpenStack APIs

Solution Overview

Output:Replica set node hostsssh-key, private ip to the init node

Input:# of replicas per shardInput:

# of nodeJS instances# MongoConfig hosts# Mongo shards hosts

Output:mongos node hostsApp EndPoint = Load-Balancer IP/path

Output:mogocfg node hosts

Input:# of config instances

resources: secgroup: type: OS::Neutron::SecurityGroup properties: name: str_replace: template: mongodb-$stackstr-secgroup params: $stackstr: get_attr: - stack-string - value rules: - protocol: icmp - protocol: tcp port_range_min: 22 port_range_max: 22 - protocol: tcp port_range_min: 27017 port_range_max: 27019

Infrastructure setup

mongo_host: type: OS::Nova::Server properties: name: str_replace: template: $stackprefix-$stackstr params: $stackprefix: get_param: stack-prefix $stackstr: get_attr: - stack-string - value Image:

get_param: image flavor: get_param: flavor security_groups: - get_param: security_group

Create Compute Instances

mongodb_peer_servers: type: "OS::Heat::ResourceGroup" properties: count: { get_param: peer_server_count } resource_def: type: { get_param: child_template } properties: server_hostname: str_replace: template: '%name%-0%index%' params: '%name%': { get_param: server_hostname } image: { get_param: image } flavor: { get_param: flavor } ssh_key: { get_resource: ssh_key } ssh_private_key: { get_attr: [ssh_key, private_key] } kitchen: { get_param: kitchen } chef_version: { get_param: chef_version }

Create MongoDB Replica Servers

server_setup: type: "OS::Heat::ChefSolo" depends_on: - mongodb_peer_servers properties: username: root private_key: { get_attr: [ssh_key, private_key] } host: { get_attr: [mongodb_peer_servers, accessIPv4, 0] } kitchen: { get_param: kitchen } chef_version: { get_param: chef_version } node: mongodb: ruby_gems: mongo: '1.12.0' bson_ext: '1.12.0' bind_ip: { get_attr: [mongodb_peer_servers, privateIPv4, 0] } use_fqdn: false replicaset_members: { get_attr: [mongodb_peer_servers, privateIPv4] } config: replset: myreplset run_list: [ "recipe[config_replset]" ]

Configure the Replica Servers

nodestack_chef_run: type: 'OS::Heat::ChefSolo' depends_on: nodestack_node properties:

... node: nodejs_app:

... deployment: id: { get_param: stack_id } app_id: nodejs run_list: ["recipe[apt]", "recipe[nodejs]", "recipe[ssh_known_hosts]", "recipe[nodejs_app]"] data_bags: nodejs: id: { get_param: stack_id } nodejs_app: password: { get_attr: [nodejs_user_password, value] } deploy_key: { get_param: deploy_key } database_url: str_replace: template: 'mongodb://%dbuser%:%dbpasswd%@%dbhostname%' params: '%dbuser%': { get_param: database_username } '%dbpasswd%': { get_param: database_user_password } '%dbhostname%': { get_param: db_server_ip }

Create NodeJS Container

Cons: ◇ OpenStack Only ◇ Software configuration is

limited◇ Lack of built-in workflow

abstraction ◇ Post deployment

orchestration is limited￭ Requires integration with other tools/ projects

Heat: Pros and Cons Pros: ◇ Native To OpenStack◇ Built-in mapping of

most OpenStack infrastructure resource types

So, what’s the Right Tool for me? ◇ Are you hell bent on containers? OpenStack? ◇ Do you have legacy workloads? ◇ Do you consider infrastructure resources as part of

the process? ◇ Do you have a heterogenous environment? ◇ Do you want the same tool to also handle post

deployment?

The Tools Presented Are Not Necessarily Mutually Exclusive!

Final Words

The Only Constant is Change

Thank YouQuestions?Find us at:

◇ Twitter @CloudifySource◇ email info@getcloudify.org