CloudStack, jclouds and Whirr!
-
Upload
andrew-bayer -
Category
Technology
-
view
3.718 -
download
1
Transcript of CloudStack, jclouds and Whirr!
Apache Java Cloud Projects!
Andrew BayerBay Area Apache CloudStack Users GroupAugust 7, 2013
Who am I?
Andrew Bayer
Build and Tools Architect at Cloudera
User of CloudStack
VP, Apache Whirr
PPMC, Apache jclouds
Bad at designing slide decks
What am I here to talk about?
How the Apache Java-based cloud projects play together!
CloudStack(of course!)jcloudsWhirr
Hold on a sec, gotta start the demo!
(it takes a while, y'see)
What's CloudStack?
...shouldn't you know that already?
Well, ok.
The project develops open source software for deploying public and private Infrastructure-as-a-Service (IaaS) clouds.
What can you do with CloudStack?
Run a cloud?
Seriously, what can't you do with CloudStack?
What's jclouds?
Java library for abstracting and interacting more directly with a plethora of compute, blobstore (i.e., S3 or Swift) and other cloud-related APIs and providers.
High-level abstractions allow you to use the same code to work with different APIs/providers.
Java interfaces directly to the various APIs let you drill down and get more specific control.
What can you do with jclouds?
Cloud automation in your Java/JVM application
Provision instancesUse object storesControl your cloudWrite a management layer on top of one or more clouds (hey, I did that!)
What's Whirr?
A set of Java libraries (built on top of jclouds) for provisioning and deploying distributed systems.
Cloud-neutral, or bring your own node
Common service API with service-specific details for configuring/provisioning
Smart defaults so you can get going easily, but easy to override
What can you do with Whirr?
Set up an Apache Hadoop cluster on your favorite cloud in 5 minutes of work.Deploy a distributed application onto existing physical hosts (yes, Whirr works with bare metal!)Write your own service to deploy your own appIntegrate all the above into a Java/JVM app
All at Apache!
CloudStack and jclouds
jclouds has support for the full CloudStack API
User, Domain and Root APIs(if something's missing, open a JIRA and we'll add it!)
jclouds compute abstraction allows you to use the same code with multiple clouds
Or you can drill down to the full CloudStack-specific API and do, well, anything!
CloudStack and jclouds
CloudStack is a first class API in jclouds
Right up there with EC2, OpenStack, and (soon) Google Compute
Provisioning, security groups, image creation one abstraction works with all these clouds
jclouds: Testing, Testing, Testing
Full battery of API tests
Unit tests to verify correctness of API calls
Live tests exercise every API call against an actual CloudStack setup
Wait, All the APIs?
You mean the Root Admin APIs too?
Creating/deleting zones, acounts, etc?
Yup.
But isn't that risky to run against a live CloudStack setup?
Ah, I've got an answer for that!
So how do you do that safely?
The CloudStack simulator, of course!
Build CloudStack, run the simulator(and the marvin scripts to configure it, etc)
Point jclouds' tests at the simulator
Test everything! Break nothing!
A bit more detail...
You do need to grab the root admin API creds, and create domain admin/normal users as well
Not all tests will actually work anything that expects to SSH into a VM or attach a disk, etc
We'll be marking simulator-friendly tests so you can run mvn clean install -Plive -Pcloudstack-simulator and just run those tests
So what's using jclouds and CloudStack now?
A good number of tools, like...
- The Jenkins jclouds plugin(for provisioning Jenkins slaves on the fly)- Pallet(config management like Chef/Puppet, in Clojure)- CloudCat(Cloudera's internal cloud management app)...and of course, Whirr!
Is Whirr like Chef or Puppet too?
Short answer? No.
Chef/Puppet/Pallet/etc are great at getting nodes to a target state, initially and/or regularly
But they're not good at dealing with a set of distributed apps that need to coordinate with each other while getting set up
Whirr, on the other hand, is.
Deployment Orchestration Engine!
That's the term I use to describe Whirr
Coordinating installation and configuration, across hosts/instances, in pre-defined stages
What services does Whirr support?
Among others:
HadoopHBaseZooKeeperCassandraChefPuppetSolrKerberos
and more...
My personal favorite: whirr-cm
(I'm admittedly a bit biased)That's Whirr support for Cloudera Manager
Deploys a fully configured CDH (Cloudera's Distribution for Apache Hadoop) cluster, including Cloudera Impala and Search, using Cloudera Manager
All in one command!
Cloudera Manager
I should probably mention a bit more about CM.
It manages, reports, configures, monitors, deploys the complete CDH Hadoop stack, with lots of handy features and integrations, all exposed via a great web UI and a REST API (which has publicly available Java and Python libraries as well).
whirr-cm works everywhere
...well, nearly everywhere.
Since it's jclouds-powered, it works with any cloud supported by jclouds.
But it does require certain things like reverse-resolvable public IPs, as do some other Whirr services.
Running whirr
Set up a config file for your cluster see the recipes dir for examples
For CloudStack, you have to specify jclouds.endpoint=http://your.server/client/api
Run bin/whirr launch-cluster config path/to/your.properties to provision/deploy
Run bin/whirr destroy-cluster config path/to/your.properties to destroy
Oh yeah, that demo...
It's whirr-cm!
Let's see how it's doing...
Another caveat
Whirr 0.8.2 (the latest release) has some problems with CloudStack
Namely, security groups/firewalls and keypairs
These will be fixed in 0.9.0, coming out this fall
Patch already available - WHIRR-725
How do I get whirr-cm?
First option: download or build Whirr, and then download or build whirr-cm see https://github.com/cloudera/whirr-cm
Second option: install Whirr from the CDH deb/rpm packages CDH 4.3.0 uses whirr-cm 1.2.
Links!
CloudStack - http://cloudstack.apache.org/
jclouds - http://jclouds.incubator.apache.org/
Whirr - http://whirr.apache.org/
whirr-cm - https://github.com/cloudera/whirr-cm
Installing Whirr from CDH
CloudCat - https://github.com/abayer/cloudcat/ (for CloudStack-specific API use with jclouds)
Thanks for listening!