How to Puppetize Google Cloud Platform - PuppetConf 2014

HOWTO:Puppetizing Google Cloud Platform

Katharina Probst, Technical Lead & Manager (Google)Ryan Coleman, Product Manager (Puppet Labs)

Agenda: September 23rd, 2014

Whirlwind tour of Google Cloud Platform

Diving into Google Compute Engine

How Puppet Labs uses Google Cloud Platform

Live Demo

1

2

3

4

For the past 15 years, Google has been building out the world’s fastest, most powerful, highest quality cloud infrastructure on the planet.

Images by Connie Zhou

Why Google Cloud Platform?

Building Products that Scale

Google Maps Gmail Google Drive YouTube

A Network that Spans the Globe

Innovating Software & Driving Technology Forward

SpannerDremelMapReduce

Big Table Colossus

2012 20132002 2004 2006 2008 2010

GFSCompute

Engine

“[Google's] ability to build, organize, and operate a huge network of servers and fiber-optic cables with an efficiency and speed that rocks physics on its heels.

This is what makes Google Google: its physical network, its thousands of fiber miles, and those many thousands of servers that, in aggregate, add up to the mother of all clouds.”

- Wired


Storage

Cloud Storage Cloud SQL Cloud Datastore

Compute

Compute EngineApp Engine

App Services

BigQuery Cloud Endpoints

Google Cloud Platform

Cloud DNS

Compute


Storage


Power Computation

App Services


Cloud DNS

Storage


Storing What You Want, However You Want

Compute


App Services


Cloud DNS

App Services


Building Robust, Intelligent Systems

Storage


Compute


Cloud DNS





Live Demo

1

2

3

4

• IaaS: VMs, Network, Storage

• Google DNA (speed, scale, reliable, secure)

• Fast Provisioning, Consistent Performance

• Enterprise Ready

• 24x7 Support

• 99.95% monthly SLA

• ISO 27001, SSAE-16 SOC 1,2,3

• Accessible Through

• Web @ https://cloud.google.com/console

• gcloud compute command-line utility

• REST API

• Partners (Commercial and FOSS)

Google Compute Engine

https://cloud.google.com/console

Launching 50 virtual machines

http://graphite-demos.appspot.com/quick-start

Block StoragePersistent Disks

Persistent Disk

10 TB

• SSD and "Standard"

• Data and root partitions

• Billed only by capacity (GB/month)

• Performance caps scale linearly with size

• Volume striping is automatic

• Differential snapshots

• Create new PDs based on snapshots

PD Use-cases

Root

Stateful root volume

User managed data volume

Root RW Data

RO Data

Instant distribution of static content

Global Snapshot and Restore

GCS

vol1 vol2 vol3

vol1.t2

vol2.t2

vol3.t3

t1 t1 t1

t1 t1 t1

t1 t1 t1

t1 t1

t1 t1

t2 t2

t2

t2

t3 t3 t3

● Point in time snapshot to Google Cloud Storage (GCS)

● Differential snapshots

● GCS global replication!

● Restore from snapshot anywhere in the world

us-central1-a

europe-west1-a

Virtual Machines

standard• For workloads with balanced CPU and memory

highmem• For workloads with higher memory requirements

highcpu• For workloads with higher CPU requirements

Shared Core Instances• For inexpensive prototyping and staging

workloads (g1-small, f1-micro)

Machine Types Machine Type Virtual CoresMemory

(GB)

n1-standard-1 1 3.75




n1-standard-16 16 60.00

n1-highmem-2 2 13.00



n1-highmem-16 16 104.00

n1-highcpu-2 2 1.80

n1-highcpu-4 4 3.60

n1-highcpu-8 8 7.20

n1-highcpu-16 16 14.40

Images

+ Custom Kernel Support(debian backports works great for docker)

Sub-hour Billing + Sustained Use Discount

1 minute granularity, 10 minute minimum

$

(30%)

Full billing cycle

Sustained Use Discount● >25% use, price reduced

● Up to 30% reduction

● No commitment / contract

Enhanced Reliability

us-central1-a

scheduled maintenance event

Live Migration• No downtime during scheduled

datacenter maintenance events

Automatic Restart• Instances automatically

restarted if subjected to system events such as hardware failure

Networking


Networking• Google's Network

• Projects are isolated private networks

• TCP, UDP, ICMP only

• Multiple private network groups and firewalls

• Tags and address ranges

• Addresses

• public: static or ephemeral

• private: ephemeral with DNS

• Routes, gateways, VPNs, and IP Forwarding

Region: us-central1

Target Pool (tp-a)

Load-balancer• Region based

• Session affinity, hashing options on tuples,

• Protocol, src ip:port, dst ip:port

• Consists of:

• Target pool: collection of instances

• Forwarding rules: protocol:port, pool

• HTTP health check: optional us-central1-b

us-central1-a

www0

www1

www2

www3

Forwarding Rules

tcp:443 ➔ tp-a

tcp:80 ➔ tp-a

Internet

http://googlecloudplatform.blogspot.com/2013/11/compute-engine-load-balancing-hits-1-million-requests-per-second.html

...and

Noteworthy• Regions

• Central United States, Europe, Asia (new!)

• Metadata

• Startup scripts

• SSH Keys

• Tags (instance/network)

• OAuth2 and Scopes

• Access other Google Cloud Platform services

• Ecosystem is growing

• Partners: RightScale, Scalr, New Relic, MongoLab, MapR, and many more...

• Open Source: Chef, Puppet, Salt, Ansible, Vagrant, Docker, CoreOS, fog, libcloud





Live Demo

1

2

3

4

Puppet DSL for Google Compute Enginehttps://forge.puppetlabs.com/puppetlabs/gce_compute

The Puppet Forgea real world (meta) example

Abstraction Layers for Infrastructure as Code

Infrastructure Layer

Role & Profile Layer

Component Layer

Primitive Layer

puppetlabs-gce_compute

custom module

puppetlabs-postgresql

core puppet type/provider





Live Demo

1

2

3

4

Demo Architecture

1 3

2 4

M

us-central1-a

us-central1-b

allo

w tc

p:80

Load

-bal

ance

r

Targ

et-p

ool

● [M]aster resides in GCE● Uses gce_compute manifest to:

○ Spin up 4 VMs○ Networking (firewall, LB)

● New nodes register with Master● Install site.pp (apache+php+mysql)● Custom index.php, connects to

Cloud SQL database

● Let's try it out!

Cloud SQL

Bring up the Compute Engine environmentpuppet apply /etc/puppetlabs/puppet/manifests/gce_up.pp

# puppet apply /etc/puppetlabs/puppet/manifests/gce_up.pp # ... # cat !$ 1 $prefix = 'pc' 2 $zonea = 'us-central1-a' 3 $zoneb = 'us-central1-b' 4 $region = 'us-central1' 5 6 gce_auth { "graphite-demos": } 7

Bring up the Compute Engine environment (cont)puppet apply /etc/puppetlabs/puppet/manifests/gce_up.pp

8 gce_firewall { "$prefix-firewall": 9 ensure => present, 10 description => 'Allow HTTP', 11 network => 'default', 12 allowed => 'tcp:80', 13 allowed_ip_sources => '0.0.0.0/0', 14 } 15 16 gce_httphealthcheck { "$prefix-http": 17 ensure => present, 18 require => Gce_instance["$prefix-agent-1", "$prefix-agent-2"...], 19 description => 'basic http health check', 20 } 21 ...


46 gce_disk { [ "$prefix-agent-2", "$prefix-agent-4" ]: 47 ensure => present, 48 size_gb => 10, 49 zone => "$zoneb", 50 source_image => 'debian-7', 51 } 52 ...


73 define create_instance_zoneb { 74 gce_instance { "${title}": 75 ensure => present, 76 zone => "$zoneb", 77 disks => "${title},boot", 78 require => Gce_disk["${title}"], 79 puppet_service => present, 80 startupscript => 'pe-simplified-agent.sh', 81 metadata => { 82 'pe_role' => 'agent', 83 'pe_master' => "$hostname", 84 'pe_version' => '3.3.2', 85 } 86 } 87 } 88 89 create_instance_zoneb { ["$prefix-agent-2", "$prefix-agent-4"]: }

The Demo Apppuppet apply /etc/puppetlabs/puppet/manifests/gce_up.pp

1 <html>2 <title>PuppetConf 2014</title>3 <body>4 <h1>PuppetConf 2014: Puppetizing Google Cloud Platform</h1>5 ...6 7 <ul>This app is being served from host: <b><%= @ hostname %> [<%= @fqdn %>, 7 <%= @ipaddress %>]</b></ul>8 <p>Please enter a Nick and Message and watch it appear below:</p>9 ...10 <?php11 $db = mysqli_connect("173.194.254.93", "puppetconf", " my_password", "puppetconf");12 ...13 ?>

cloud.google.com• Try out Google Cloud Platform with a $500 Credit

• Visit https://cloud.google.com/starterpack and use promo code puppet14-con

• Want more?

• Try the demo on your own at https://bit.ly/puppet-gce-demo

• or see http://googlecloudplatform.github.io/compute-video-demo-puppet

• We welcome your help and feedback to make Puppet + GCE even better!

• Find us at [email protected] or

https://github.com/puppetlabs/puppetlabs-gce_compute

https://cloud.google.com/starterpack

https://www.google.com/url?q=https%3A%2F%2Fbit.ly%2Fpuppet-gce-demo&sa=D&sntz=1&usg=AFQjCNFjHdyo3569PNkUPqIQOG9eQ3a2CQ

http://www.google.com/url?q=http%3A%2F%2Fgooglecloudplatform.github.io%2Fcompute-video-demo-puppet&sa=D&sntz=1&usg=AFQjCNHGfW_7w6t1jr2gLslW3efGsqFYEg

https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fpuppetlabs%2Fpuppetlabs-gce_compute&sa=D&sntz=1&usg=AFQjCNF1rD_6VfbcO4iDdSUUuX02N7woow

https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fpuppetlabs%2Fpuppetlabs-gce_compute&sa=D&sntz=1&usg=AFQjCNF1rD_6VfbcO4iDdSUUuX02N7woow

How to Puppetize Google Cloud Platform - PuppetConf 2014

Technology

Transcript of How to Puppetize Google Cloud Platform - PuppetConf 2014