Post on 17-May-2015
description
02/01/11
Presented to JClouds Meetup @ Twitter’s HQToni Batchelli, Consultant and Entrepreneur wannabe
tbatchelli@gmail.com, @tbatchelli, @disclojureFebruary 2, 2011
Complex Infrastructure Made Easy™
Agile Development at GoGrid with Pallet and JClouds
GoGrid’s Challenges
GoGrid is a cloud provider, building their own software
Developing and testing their cloud management software is complex. They need to create and maintain realistic environments in order to load them with their latest software builds
~10 different servers types~20 different services
The problem?
Current time to setup a development/QA environment is 2 days• Mostly manual work
Building the environments manually is very error prone
A high level of interconnectivity between services
Goal
To be able to create development and testing environments dynamically, in a
customizable way, and on cloud infrastructure
A Cloud in a Cloud
Additional ChallengesEnvironments need to be verified as fully functional before deploying software on them
Must support multiple locations
Must be able to scale dynamically
Must support concurrent development• Different feature branches
Must manage/version non-software components in a lockstep with the software:• Network configuration• System-level configurations• Application-level configurations
02/01/11
tools
Abstraction layer on resource management
If you code your infrastructure management code to the jclouds API, you can port your code to all cloud providers
JVM-based
Provides access to GoGrid’s infrastructure by abstracting the communication GoGrid’s Cloud API
Uses JClouds to create and manage nodes
Provisions nodes with software and configuration
Verifies node and service correct operation
Operates the services in the nodes
Nodes are grouped by groups (formerly known as ‘tags’). All nodes in a group are equivalent
Clojure
Pallet is built on ClojureClojure is a JVM-based lisp with superb integration with Java and concurrency
Can use any existing Java library (Tomcat, jdbc, etc...)
Provides a high level of abstraction, which is what is needed in this case
Growing community: there are 7 books on Clojure either published or in preparation
02/01/11
how?
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
ServiceServiceService
ServiceService
Service
Service Service
Service ServiceService
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
ServiceServiceService
ServiceService
Service
Service Service
Service ServiceService
jclouds
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
ServiceServiceService
ServiceService
Service
Service Service
Service ServiceService
gimmenodes!
jclouds
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
ServiceServiceService
ServiceService
Service
Service Service
Service ServiceService
gimmenodes!
jclouds
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
ServiceServiceService
ServiceService
Service
Service Service
Service ServiceService
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
ServiceServiceService
ServiceService
Service
Service Service
Service ServiceService
pallet
pallet
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
ServiceServiceService
ServiceService
Service
Service Service
Service ServiceService
do stuff!
pallet
pallet
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
ServiceServiceService
ServiceService
Service
Service Service
Service ServiceService
do stuff!
pallet
pallet
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
jclouds
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
make mea dev cloud!
jclouds
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
make mea dev cloud!
jclouds
ServiceService
Service
ServiceServiceService
ServiceServiceService
Service Service
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
ServiceService
Service
ServiceServiceService
ServiceServiceService
Service Service
pallet
pallet
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
install cloudinfrastructure!
ServiceService
Service
ServiceServiceService
ServiceServiceService
Service Service
pallet
pallet
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
install cloudinfrastructure!
ServiceService
Service
ServiceServiceService
ServiceServiceService
Service Service
pallet
pallet
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
ServiceService
Service
ServiceServiceService
ServiceServiceService
Service Service
pallet
pallet
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
is all workingfine?
ServiceService
Service
ServiceServiceService
ServiceServiceService
Service Service
pallet
pallet
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
is all workingfine?
ServiceService
Service
ServiceServiceService
ServiceServiceService
Service Service
pallet
pallet
OK!
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
ServiceService
Service
ServiceServiceService
ServiceServiceService
Service Service
pallet
pallet
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
deploynew build!
ServiceService
Service
ServiceServiceService
ServiceServiceService
Service Service
pallet
pallet
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
ServiceService
Service
ServiceServiceService
ServiceServiceService
Service Service
pallet
pallet
deploynew build!
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
ServiceService
Service
ServiceServiceService
ServiceServiceService
Service Service
pallet
pallet
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
deploynew build!
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
ServiceService
Service
ServiceServiceService
ServiceServiceService
Service ServiceGoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
awesome!
ServiceService
Service
ServiceServiceService
ServiceServiceService
Service ServiceGoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
ServiceService
Service
ServiceServiceService
ServiceServiceService
Service ServiceGoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
ServiceService
Service
ServiceServiceService
ServiceServiceService
Service ServiceGoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PIjclouds
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
let’s test:gimme nodes!
ServiceService
Service
ServiceServiceService
ServiceServiceService
Service ServiceGoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PIjclouds
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
let’s test:gimme nodes!
ServiceService
Service
ServiceServiceService
ServiceServiceService
Service ServiceGoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PIjclouds
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
ServiceService
Service
ServiceServiceService
ServiceServiceService
Service ServiceGoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
jclouds
ServiceService
Service
ServiceServiceService
ServiceServiceService
Service ServiceGoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
I’m done,thanks!
jclouds
ServiceService
Service
ServiceServiceService
ServiceServiceService
Service ServiceGoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
I’m done,thanks!
jclouds
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
environmentfor branch A?
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
branch A
environmentfor branch A?
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
branch A
environmentfor branch A?
environmentfor branch B?
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
branch A
environmentfor branch A?
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
branch B
environmentfor branch B?
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
branch A
environmentfor branch A?
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
branch B
environmentfor branch B?
environmentfor QA please?
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
branch A
environmentfor branch A?
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
branch B
environmentfor branch B?
GoGrid Cloud Infrastructure
GoG
rid
Clo
ud A
PI
QA
environmentfor QA please?
02/01/11
development
GoGrid’s Development Process
Create a feature branch
Create an environment (cloud infrastructure) for this branch
Verify new environment (end-to-end tests)
Automatically build and deploy the new software every time new code is committed to the branch.
Destroy environment and free resources when done
An Environment
Usually one environment for active development branch
Each environment is specified by:• The development branch it tracks• Service types and node count for each service type• Network configuration• External services/DB provisioning
Development Environments
Each development environment tracks a development feature branch
Each environment has a node with Hudson CI, configured to track the environment’s branch
When new code is pushed to the development branch, the environment’s Hudson server builds the software and updates the environment with it.
02/01/11
more on pallet
Why Pallet?
There are well known solutions for node provisioning in the cloud, why pallet?• It integrates with infrastructure management• Service interconnections are first class entities in
the framework• Scalable programming model• Integrates well with SCM and with other OPS tools• Comprehensive support for all development cycles
✓No central server
✓No agents on nodes
✓Works with standard and custom images
✓First-class support for complex multi-service configurations
✓All node configuration done remotely via generated scripts over SSH
✓DSL for generating scripts, resources
✓Testable
crates
Pallet’s basic configuration unit
A crate is a grouping of functions around a service
Functions usually:• manipulate/create internal representations• result in resources deployed or scripts being run in
the target nodes
proxied
bootstrap
automatedadmin user
tomcat
java tomcat
tomcatdeploy
reverse proxy
haproxy
Custom Crates
Standard Crates
A Node(core/defnode proxied "Basic web app, served by tomcat" {:os-‐family :ubuntu :os-‐description-‐matches "10.04" :inbound-‐ports [8080 22]} :bootstrap (resource/phase (crates/bootstrap)) :configure (resource/phase (crates/tomcat)) :deploy (resource/phase
(crates/tomcat-‐deploy "webapp.war")) :restart-‐tomcat (resource/phase (service/service
"tomcat6":action :restart)))
jclouds section of a node(core/defnode proxied "Basic web app, served by tomcat" {:os-‐family :ubuntu :os-‐description-‐matches "10.04" :inbound-‐ports [8080 22]} :bootstrap (resource/phase (crates/bootstrap)) :configure (resource/phase (crates/tomcat)) :deploy (resource/phase
(crates/tomcat-‐deploy "webapp.war")) :restart-‐tomcat (resource/phase (service/service
"tomcat6":action :restart)))
Execution Phases(core/defnode proxied "Basic web app, served by tomcat" {:os-‐family :ubuntu :os-‐description-‐matches "10.04" :inbound-‐ports [8080 22]} :bootstrap (resource/phase (crates/bootstrap)) :configure (resource/phase (crates/tomcat)) :deploy (resource/phase
(crates/tomcat-‐deploy "webapp.war")) :restart-‐tomcat (resource/phase (service/service
"tomcat6" :action :restart)))
Cloud Infrastructure
Pallet
Crates
Node definitions
Clo
ud A
PI
webapp webapp webapp
proxy proxy
DB DB
Cloud Infrastructure
Pallet
Crates
Node definitions
Clo
ud A
PI
webapp webapp webapp
proxy proxy
DB DB
Cloud Infrastructure
Pallet
Crates
Node definitions
Clo
ud A
PI
webapp webapp webapp
proxy proxy
DB DB
Cloud Infrastructure
Pallet
Crates
Node definitions
Clo
ud A
PI
webapp webapp webapp
proxy proxy
DB DB
jclouds
Cloud Infrastructure
Pallet
Crates
Node definitions
Clo
ud A
PI
webapp webapp webapp
proxy proxy
DB DB
proxy proxy
jclouds
Cloud Infrastructure
Pallet
Crates
Node definitions
Clo
ud A
PI
webapp webapp webapp
proxy proxy
DB DB
proxy proxy
webapp webapp webappjclouds
Cloud Infrastructure
Pallet
Crates
Node definitions
Clo
ud A
PI
webapp webapp webapp
proxy proxy
DB DB
proxy proxy
webapp webapp webapp
DB DB
jclouds
Cloud Infrastructure
Pallet
Crates
Node definitions
Clo
ud A
PI
webapp webapp webapp
proxy proxy
DB DB
proxy proxy
webapp webapp webapp
DB DB
Cloud Infrastructure
Pallet
Crates
Node definitions
Clo
ud A
PI
webapp webapp webapp
proxy proxy
DB DB
proxy proxy
webapp webapp webapp
DB DB
squid squid
Cloud Infrastructure
Pallet
Crates
Node definitions
Clo
ud A
PI
webapp webapp webapp
proxy proxy
DB DB
proxy proxy
webapp webapp webapp
DB DB
squid squid
Cloud Infrastructure
Pallet
Crates
Node definitions
Clo
ud A
PI
webapp webapp webapp
proxy proxy
DB DB
proxy proxy
webapp webapp webapp
DB DB
squid squid
tomcat tomcattomcat
Cloud Infrastructure
Pallet
Crates
Node definitions
Clo
ud A
PI
webapp webapp webapp
proxy proxy
DB DB
proxy proxy
webapp webapp webapp
DB DB
squid squid
tomcat tomcattomcat
Cloud Infrastructure
Pallet
Crates
Node definitions
Clo
ud A
PI
webapp webapp webapp
proxy proxy
DB DB
proxy proxy
webapp webapp webapp
DB DB
squid squid
tomcat tomcattomcat
mysql mysql
How are systems interconnected?
Phase 1• Retrieve and/or create needed nodes• For each node in each group: run phases for crates
anda) Collect configuration infob) Schedule configuration actions to be performed
on node
Phase 2• Generate and execute scripts and resources
02/01/11
how does this look today?
(def branch-1-environment {:branch "http://svn....../branches/branch-1" :compute-service {:provider "gogrid" :identity "<username>" :credential "<password>" :endpoint "http://10...:8080/api/"} :db-name "branch-1-db" :db-properties env-network-properties :pre-bootstrap-fn #'setup-environment-db :bootstrap-tags {com.gogrid.nodes.services/LDAP 1 com.gogrid.nodes.services/Cache 1} :bootstrap-phases [:restart-cache :restart-LDAP] :verify-phases [:verify]})
Environment Definition
user => (start branch-1-environment)
user => (verify branch-1-environment)
user => (destroy branch-1-environment)
Operation
It’s a newcomer in the field.• Documentation needs work• Untested support for some combinations of flavors
and versions of Linux
Steep learning curve, a new language
OMG! It’s A Lisp!
Challenge with Pallet
Some pointers...
http://jclouds.org Adrian Cole et al.
http://palletops.com Hugo Duncan et al.
http://gogrid.com For more info on GoGrid services please contact:
Paul LappasVP Engineering & Co-Founder of GoGridpaul@gogrid.com
GoGrid is hiring devops and SW engineers for their SF office (contact Paul too)
02/01/11
Toni Batchellitbatchelli@gmail.com
@tbatchelli@disclojure
Questions?