Vertically Scaled Design Patterns

Jeff Malnick & Paul AmbrosiniSystems Engineering @ SRC:CLR

srcclr.com | @srcclr

How this is going down:

srcclr.com | @srcclr

● History & Evolution

● Principles

● Notes & Examples

● Demo

History & Evolution

srcclr.com | @srcclr

In the beginning there was

srcclr.com | @srcclr

● AWS● ELB● Ansible ● SCP● Tomcat

● We all start somewhere

Then there was

srcclr.com | @srcclr

srcclr.com | @srcclr

Paul heroically rolled it out in 2 weeks

Massive Gains

srcclr.com | @srcclr

● Huge improvement

● Split up Frontend and Backend Code

● Teams deploying separately

● Productivity went up dramatically

Wasn’t perfect

srcclr.com | @srcclr

● Not fully dynamic

● CI hooks not perfect

● Cookbook merging*

● Service health management lacking

Then we migrated...

srcclr.com | @srcclr

● *hired Jeff○ ‘ops role’○ or ‘devops’

We got a shiny new CM system.

srcclr.com | @srcclr

"Type" : "AWS::EC2::Instance",

"Properties" : {

"KeyName" : "secure",

"SecurityGroupIds" : ["sg-1234abcd","sg-5678efg"],

"SubnetId" : "subnet-1234abcd",

"UserData" : {

"Fn::Base64" : {

"Fn::Join" : ["", [

"#!/bin/sh -v\n",

"ROLE='qa_nginx_static'\n",

"MASTER='https://puppet.srcclr.com'\n",

"D_ROLE=$(echo $ROLE | tr '_' '-')\n",

"D_IP=$(hostname -I | tr '.' '-' | cut -d' ' -f1)\n",

"HOSTNAME=\"$(echo $D_ROLE)-$(echo $D_IP)\"\n",

"echo \"127.0.0.1 $HOSTNAME\" >> /etc/hosts",

"apt-get update\n",

"echo $HOSTNAME > /etc/hostname\n",

"/bin/hostname $HOSTNAME\n",

"echo $ROLE > /etc/role\n",

"sudo curl -k $MASTER:8140/packages/current/install.bash | sudo bash\n"

]]}

Cloud Formation for Bare Metal

Continuous Integration

srcclr.com | @srcclr

srcclr.com | @srcclr

# Executed in $WORKSPACE on jenkins

VERSION=$(some_mvn_command)

curl \

-X POST \

-d@- \

puppet.myorg.com:1015/deploy <\<EOF

{

"service": "analytics",

"version": "$VERSION",

"environment": "qa",

"role": "qa_service"

}

EOF

How we do CI

New CM was better but...

srcclr.com | @srcclr

Shameless use of hooks

srcclr.com | @srcclr

include http

http::listener {‘kick_haproxy’:

port => '1234',

routes => {

'kick_haproxy' => {

'method' => 'get',

'command' => "su - peadmin -c 'mco puppet runonce -F

role == haproxy_internal'"

},

“Service Discovery”

srcclr.com | @srcclr

http { 'kick_haproxy':

ensure => get,

port => '1234',

fqdn => 'puppet.myorg.com'

}

srcclr.com | @srcclr

sc_services::service { 'librarian':

deploy_stage => 'production',

version => $version,

enable_newrelic_apm => true,

loadbalancer_role => 'production_internal',

xmx_setting => '1024',

xms_setting => '1024',

proc_opts => {

'1' => {

'env_profile' => 'prod',

'port' => '11100',

'mgmt_port' => '11110',

'override' => $properties_file,

},

'2' => {

'env_profile' => 'prod',

'port' => '11101',

'mgmt_port' => '11111',

'override' => $properties_file,

}

}

Still static...

srcclr.com | @srcclr

sc_services::service { 'librarian':

deploy_stage => 'app',

version => $version,

enable_newrelic_apm => true,

loadbalancer_role => 'production_internal',

xmx_setting => '1024',

xms_setting => '1024',

proc_opts => {

'1' => {

'env_profile' => 'app',

'port' => '11100',

'mgmt_port' => '11110',

'override' => $properties_file,

},

'2' => {

'env_profile' => 'app',

'port' => '11101',

'mgmt_port' => '11111',

'override' => $properties_file,

}

}

Still static...

{Statically

assigning ports!

srcclr.com | @srcclr

sc_services::service { 'librarian':

deploy_stage => 'app',

version => $version,

enable_newrelic_apm => true,

loadbalancer_role => 'production_internal',

xmx_setting => '1024',

xms_setting => '1024',

proc_opts => {

'1' => {

'env_profile' => 'app',

'port' => '11100',

'mgmt_port' => '11110',

'override' => $properties_file,

},

'2' => {

'env_profile' => 'app',

'port' => '11101',

'mgmt_port' => '11111',

'override' => $properties_file,

}

}

Still static...

What S3 Bucketis this Jar from?

srcclr.com | @srcclr

sc_services::service { 'librarian':

deploy_stage => 'app',

version => $version,

enable_newrelic_apm => true,

loadbalancer_role => 'production_internal',

xmx_setting => '1024',

xms_setting => '1024',

proc_opts => {

'1' => {

'env_profile' => 'app',

'port' => '11100',

'mgmt_port' => '11110',

'override' => $properties_file,

},

'2' => {

'env_profile' => 'app',

'port' => '11101',

'mgmt_port' => '11111',

'override' => $properties_file,

}

}

Still static...

“Service Discovery”

for HaProxy

srcclr.com | @srcclr

sc_services::service { 'librarian':

deploy_stage => 'app',

version => $version,

enable_newrelic_apm => true,

loadbalancer_role => 'production_internal',

xmx_setting => '1024',

xms_setting => '1024',

proc_opts => {

'1' => {

'env_profile' => 'app',

'port' => '11100',

'mgmt_port' => '11110',

'override' => $properties_file,

},

'2' => {

'env_profile' => 'app',

'port' => '11101',

'mgmt_port' => '11111',

'override' => $properties_file,

}

}

Still static...Service statically

scheduled to execute on a

node classified by Puppet

“Scaling”

srcclr.com | @srcclr

$ports = {

'librarian' => {

'svc' => '20060-20070',

'mgmt' => '20071-20080',

},

'notifications' => {

'svc' => '20000-20010',

'mgmt' => '20011-20020',

},

'cloud_agent' => {

'svc' => '31040-31050',

'mgmt' => '31051-31060',

},

'search' => {

'svc' => '41140-41150',

'mgmt' => '41151-41160',

},

“Scaling”

srcclr.com | @srcclr

$ports = {

'librarian' => {

'svc' => '20060-20070',

'mgmt' => '20071-20080',

},

'notifications' => {

'svc' => '20000-20010',

'mgmt' => '20011-20020',

},

'cloud_agent' => {

'svc' => '31040-31050',

'mgmt' => '31051-31060',

},

'search' => {

'svc' => '41140-41150',

'mgmt' => '41151-41160',

}

...

{ Scaling to maximum of 10 instances per box.

Recap

srcclr.com | @srcclr

● CM statically schedules services to run on node

● Can scale to maximum number of known ports

● This does “scale” vertically & horizontally...

Automation Principles

A Standard Factory...

srcclr.com | @srcclr

● Bare Metal Provisioning

● Configuration Management

● Remote Execution Framework

Software Principles

The Monolith…

srcclr.com | @srcclr

As a framework it is:

srcclr.com | @srcclr

● Hard to diagnose what is broken

● One part breaks, the entire thing breaks

● Unwieldy code bases

● Works for some, wasn’t for us

The Monolith Factory

srcclr.com | @srcclr

● Statically schedules services

● Doesn’t require service discovery

The Micro-Service

srcclr.com | @srcclr

As a framework it is:

srcclr.com | @srcclr

● Atomic services

● Easy(er) to figure out what is broken

● Scale or add individual pieces

The Micro-Services Factory

srcclr.com | @srcclr

● Dynamic task scheduling

● Service discovery required

The Scaling Problem...

srcclr.com | @srcclr

● Service ‘x’ requires known port assignments

● Service ‘x’ configuration changes across env’s

● Service ‘x’ should be able to run on a box with other instances of Service ‘x’ as well as Service ‘y’ and Service ‘z’

The Vertical Component

srcclr.com | @srcclr

● Start locally, on a single box

● Find a solution that works vertically, master it

● Once the vertical solution is found, move horizontally

Dynamic Task Scheduling

srcclr.com | @srcclr

● Purpose built remote execution framework

What we really need...

srcclr.com | @srcclr

● Purpose built remote execution framework

● … micro services are like streakers…

They don’t care where they’re running, they just want to be exposed.

srcclr.com | @srcclr

● Dynamic tasks scheduling

● … micro services are like streaker`

What we really need...

srcclr.com | @srcclr

● Purpose built systems

● … CM does CM; SD does SD; divide and conquer

Configuration Management

srcclr.com | @srcclr

● Is still important!

● Is still necessary!

srcclr.com | @srcclr

Our Remote Execution Framework

Our Initial Deployment

srcclr.com | @srcclr

● Apache Mesos

● Marathon

"Type" : "AWS::EC2::Instance",

"Properties" : {

"KeyName" : "secure",

"SecurityGroupIds" : ["sg-1234abcd","sg-5678efg"],

"SubnetId" : "subnet-1234abcd",

"UserData" : {

"Fn::Base64" : {

"Fn::Join" : ["", [

"#!/bin/sh -v\n",

"ROLE='qa_nginx_static'\n",

"MASTER='https://puppet.srcclr.com'\n",

"D_ROLE=$(echo $ROLE | tr '_' '-')\n",

"D_IP=$(hostname -I | tr '.' '-' | cut -d' ' -f1)\n",

"HOSTNAME=\"$(echo $D_ROLE)-$(echo $D_IP)\"\n",

"echo \"127.0.0.1 $HOSTNAME\" >> /etc/hosts",

"apt-get update\n",

"echo $HOSTNAME > /etc/hostname\n",

"/bin/hostname $HOSTNAME\n",

"echo $ROLE > /etc/role\n",

"sudo curl -k $MASTER:8140/packages/current/install.bash | sudo bash\n"

]]}

Design Pattern : Bare Metal

class roles::mesos_master { include ::profiles::mesos::basic include ::profiles::mesos::master include ::profiles::docker::marathon

include ::profiles::haproxy::marathon_template }

Design Pattern : Config Mgmt

Design Pattern : Application Layer

Mesos for Layer 7:- Deploy application layer via Docker - Ensure service ‘x’ is running ‘n’ number of

times

{ "container": { "type": "DOCKER", "docker": { "forcePullImage": true, "image": "malnick/microbrew",

Design Pattern : Service Discovery Marathon for Service Discovery/Deployment:- API to Apache Mesos- Queried for IP/Port assignments of layer 7

resources to configure haproxy

LIVE DEMO

Are you sure you want to do this?

Pray to the demo gods

Jeff Malnick Paul Ambrosini malnick at gmail dot com

github.com/malnick

@malnick

technoblogic.iosrcclr.com | @srcclr

c at c4 dot vcpaul at srcclr dot com

github.com/cl4r1ty

@cl4r1ty

c4.vc