Cloudops fundamentalsSystem Management

Bret PiattRackspace Hosting

Static configuration, meet the dodo birdSevers "appear" based on demand..Servers "disappear" and it isn't a problem "sometimes"..

Management systems must talkto the cloud APIs. The source of truth is no longer "the Visio"..

Diagrams now show functional elements, not specific systems

"Instant" provisioning is not instantTo provision you have to get data to the new node....

5GB takes 7 minutes on a 100Mbps network..5TB takes 12 hours on a 1Gbps network..5PB takes a truck ... or 52 days on a 10Gbps network!

So what do I do with all of my tools?Many tools end up in the garbage..

..unless they add (or you write) a dynamic configurationmodule..

..Devops, you may need to code not just script..

Provision by API, configure by hand?!#?

Chef

Puppet

Dev & Ops, sitting in a treeYour revision control system is not just for source code...

Use it to track all system configuration files...

Who hasn't had a dream about the daythey can see every change of a

configuration file on a server over the entire lifespan?

Meet Pallet, your new cloud creating friendChef and Puppet are great for configuring systems......but we have to first... have systems to provision!

With Pallet you can create & configure systems in a management shell..;; We can create a node, by specifying a name tag and a template. ;; webserver-template is a vector specifying features we want in ;; our image. (start-node context :webserver webserver-template)

;; At this point we can manage instance counts as a map. ;; e.g ensure that we have two webserver nodes (with-node-templates templates (converge context {:webserver 2}))

;; Images are configured differently between clouds and os's. ;; We might want to update our machines to use the latest ;; package manager. pallet has a couple of templates, and ;; you can add your own, see resources/bootstrap. Templates ;; have a default implementation, but can be specialised ;; for a given tag or operating system family. (with-node-templates templates (converge context {:webserver 1} (bootstrap-with (bootstrap-template :ensure-resolve) (bootstrap-template :update-pkg-mgr))))

Get it right once, never make a mistake againEmbrace TDD (Test Driven Design) in Ops..

Get the requirements from the business..1. Maintain 1,000,000 open connections2. Process 1,000 events per second3. Run for 1 week without errors

Go build your tests..

Now hack the system until you have a configuration that passes #1 and #2, then let the tests run until it passes #3..

..and since you built it with Chef/Puppet it is reusable!

Let the software work for youContinuous Integration (CI) is for operations too!

Ops needs a development world view..

The job isn't done when, "Everything is working normal."

Continually try to improve configurations in the sand box..

..using TDD go from staging to production at improvement milestones or a "sprint" release cycle...agile operations!

"Those lazy Ops guys.."Unit tests now becomeapp monitoring configuration files..

..eliminate the, "Our testsuite doesn't show theerror you see in production" conflicts -- maintain a single suite.

def test_sign_up visit "/" click_link "Sign up"fill_in "Email", :with => "good@example.com"select "Free account"click_button "Register" ... end

http://wiki.github.com/brynary/webrat/

The moral of the story, "Ops..".. is more programatic as APIs replace long lead time steps.

..needs to leverage development best practices.

..is not about just "keeping the lights on."

..is about optimizing everything below the app.

..must change the processes in use today.

..is more technical now than ever before.

..is becoming "software development" at the system level.

Questions?

Twitter: @bpiattbret.piatt@rackspace.com