Devops Metrics
-
Upload
patrick-debois -
Category
Technology
-
view
13.171 -
download
1
description
Transcript of Devops Metrics
DEVOPS METRICS
ANDREW SHAFER - PATRICK DEBOIS - ISRAEL GATThursday 16 June 2011
http://www.vectronicsappleworld.com/collection/articlepics/joysticks/snap23.jpg
Patrick Debois
Jedi
@patrickdebois
http://jedi.be/blog
Thursday 16 June 2011
Israel Gat
Cutter Consortium
@agileexec
http://theagileexecutive.com
Thursday 16 June 2011
Andrew Shafer - Rabble Rouser - Cloudscaling
@littleidea - http://stochasticresonance.wordpress.comThursday 16 June 2011
http://www.flickr.com/photos/quinnanya/3588989219
Thursday 16 June 2011
http://www.flickr.com/photos/zen/3233023066/
Thursday 16 June 2011
To Comparehttp://www.flickr.com/photos/thebusybrain/2492945625/
Thursday 16 June 2011
http://www.flickr.com/photos/topshampatti/219953012/
Thursday 16 June 2011
ToUnderstand
Thursday 16 June 2011
To Protect
Thursday 16 June 2011
To Improvehttp://www.flickr.com/photos/vizzzual-dot-com/2655969483/
Thursday 16 June 2011
Describing a Metric
http://www.flickr.com/photos/chandramarsono/4324373384/
Thursday 16 June 2011
Describing a Metric
•basis of measurement
http://www.flickr.com/photos/chandramarsono/4324373384/
Thursday 16 June 2011
Describing a Metric
•basis of measurement•assumptions
http://www.flickr.com/photos/chandramarsono/4324373384/
Thursday 16 June 2011
Describing a Metric
•basis of measurement•assumptions•level and usage
http://www.flickr.com/photos/chandramarsono/4324373384/
Thursday 16 June 2011
Describing a Metric
•basis of measurement•assumptions•level and usage•desired trend
http://www.flickr.com/photos/chandramarsono/4324373384/
Thursday 16 June 2011
Describing a Metric
•basis of measurement•assumptions•level and usage•desired trend•when to use
http://www.flickr.com/photos/chandramarsono/4324373384/
Thursday 16 June 2011
Describing a Metric
•basis of measurement•assumptions•level and usage•desired trend•when to use•when to stop
http://www.flickr.com/photos/chandramarsono/4324373384/
Thursday 16 June 2011
Describing a Metric
•basis of measurement•assumptions•level and usage•desired trend•when to use•when to stop•how to game
http://www.flickr.com/photos/chandramarsono/4324373384/
Thursday 16 June 2011
Describing a Metric
•basis of measurement•assumptions•level and usage•desired trend•when to use•when to stop•how to game•warnings
http://www.flickr.com/photos/chandramarsono/4324373384/
Thursday 16 June 2011
Beware of Vanity Metrics
UNACTIONABLE
INACCESSIBLE
UNAUDITABLEhttp://www.flickr.com/photos/centralasian/5829549813/
Thursday 16 June 2011
Red is good , Green is worthless
• http://www.reliableplant.com/Read/21305/lean-metrics-red-is-good,-green-is-worthless
Thursday 16 June 2011
Thursday 16 June 2011
“In truth, a good case could be made that if your knowledge is meagre and unsatisfactory, the last thing in the world you should do is make measurements; the chance is negligible that you will measure the right things accidentally.”
http://stochasticresonance.wordpress.com/2008/08/03/miles-per-gallon/
George Miller
Thursday 16 June 2011
What we can learn from monitoringhttp://www.experimentalistsanonymous.com/stuff/paralleluniverse/images/2edmoneyshot.jpg
Thursday 16 June 2011
Level 0 - Individual Resource Thursday 16 June 2011
Availability
Network, Memory, CPUProcesses, Port,SSH Login
Thursday 16 June 2011
Availability
Network, Memory, CPUProcesses, Port,SSH Login
Thursday 16 June 2011
Capacity
Network, Memory, CPUProcesses, Port,SSH Login
Thursday 16 June 2011
Capacity
Network, Memory, CPUProcesses, Port,SSH Login
Thursday 16 June 2011
Health
http://www.geocities.jp/nikonikocalendar/index_en.html
Thresholds
http://www.smiley-faces.org/wallpaper/smiley-face-wallpaper-001.jpg
Thursday 16 June 2011
Health
http://www.geocities.jp/nikonikocalendar/index_en.html
Thresholds
http://www.smiley-faces.org/wallpaper/smiley-face-wallpaper-001.jpg
Thursday 16 June 2011
http://www.slideshare.net/krivitsky/agile-metrics-presentation-654259
Measurement Metric
Diagnostic
Pete Behrens, Trail Ridge Consulting
Does itcontribute to
thebottom line?
Metric versus Diagnostic
Yes
No
Thursday 16 June 2011
Level 1 - Services LevelThursday 16 June 2011
Performance
SSH, HTTP, NTP
Thursday 16 June 2011
Performance
SSH, HTTP, NTP
# Servers/Admin
Thursday 16 June 2011
Performance
# Servers/Admin
# Line of Code/Developer
SSH, HTTP, NTP
Thursday 16 June 2011
“Gameable”
http://kithicor.org/wp/wp-content/uploads/2010/01/gamerEvolution.jpg
Thursday 16 June 2011
Level 2 - Team LevelThursday 16 June 2011
Aggregation
WEB1
LB-WEB
WEB2
WEB3
Thursday 16 June 2011
Aggregation
DEV
DEV Project Team
DEV DEV
Thursday 16 June 2011
Burn up chart
Thursday 16 June 2011
Team Utilization
Thursday 16 June 2011
“Velocity also has a direction component”
http://www.randomstew.com/wp-content/uploads/mooneye.jpg
Thursday 16 June 2011
What about efficiency?http://www.flickr.com/photos/jdiggans/92981578/
Thursday 16 June 2011
http://sievertschreiber.files.wordpress.com/2009/12/good-code-is-measured-in-wtf-per-minute.jpg
Thursday 16 June 2011
Incentives & Behaviorshttp://www.flickr.com/photos/turtlemadness/2439050717/
Thursday 16 June 2011
http://www.xgn.nl/images/upload/20081020225036.jpg http://www.kawaiikakkoiisugoi.com/wp-content/uploads/2011/03/pacman-screen.jpg
OperationsDevelopment
Level 3 - Integration LevelThursday 16 June 2011
Integration
DBWEB APP
Thursday 16 June 2011
Collaboration
# of deploys/ day
OPSDEV
http://www.sage.org/lists/sage-members-archive/2001/msg01208.html http://www.datacenterknowledge.com/archives/2009/12/30/how-many-servers-can-one-admin-manage/ http://fc04.deviantart.net/fs70/f/2010/108/2/3/Pac_Man_Widescreen_by_molotov_arts.jpg
QA
Thursday 16 June 2011
Counting Collaboration
• http://www.nature.com/news/2010/101013/full/news.2010.538.html
More interaction doesn’t mean a better party
Thursday 16 June 2011
Measuring Collaboration?
http:///www.healthylarimer.org/documents/PlayNice.ppt“Play Nice Study”Thursday 16 June 2011
Levels Collaboration
Cooperation
Coordination
Coalition
Collaboration
Thursday 16 June 2011
Cooperation
provide information
somewhat defined roles
formal communication
decisions made independently
Thursday 16 June 2011
Coordination•share information & resources
•defined roles
•frequent communication
•some shared decision making
Thursday 16 June 2011
Coalition
•share ideas
•share resources
•frequent & prioritized communication
•all members have a vote on decisions
Thursday 16 June 2011
Collaboration
•members belong to one system
•frequent communication characterized by mutual trust
•consensus on all decisions
Thursday 16 June 2011
http://www.au.af.mil/au/awc/awcgate/navy/onr_metrics_on_collab.pdf
Knowledge sharing
Thursday 16 June 2011
Team Resilience
Truck Factor10th Floor Test
http://www.kitchensoap.com/
Resilience should be a 4th management objective,
alongside Better/Faster/Cheaper John Allspaw
The number of people on your team who have to be hit with a truck before the project is in serious trouble
to take a random machine in your infrastructure , drop it from the 10th floor, and be able to restore your infrastructure to working within 5-10 minute
Thursday 16 June 2011
Black Hole Theory
http://www.collaborate.com/blog/2010/01/28/musing-metrics-collaboration
Absorbs light, to see the effects you need to look at nearby
objects
Thursday 16 June 2011
http://communityandbusiness.groupsite.com/group/Collaboration20/post/measuring-the-value-of-collaboation
• Saving time or Money (tangible)
• Increase Quality (tangible... but less so)
• Innovating and/or providing decision support (tangible but less than quality)
• Easing access to and interactions with subject-matter experts (intangible)
Quest for proof
Thursday 16 June 2011
http://socialconsulting.wordpress.com/2009/02/24/measuring-collaboration/
Quality Improvement
Tune existing serversAdd more servers
Process Improvement
Easier More Difficult!!
More vs Better Widgets
Thursday 16 June 2011
Collaboration Challenge
“Maximize
Team Utility”
Thursday 16 June 2011
Level 4 - Business LevelThursday 16 June 2011
AHA KA-CHING!
Thursday 16 June 2011
http://www.flickr.com/photos/hanekomu/4481674104/
Return on Investment
Thursday 16 June 2011
Inventory (WIP)
Thursday 16 June 2011
Time to Market
http://stefanroock.wordpress.com/2010/03/02/kanban-definition-of-lead-time-and-cycle-time/
Thursday 16 June 2011
http://museumdosgames.blogspot.com/2010/04/game-over.html
Mean Time Between Failure
Thursday 16 June 2011
Mean Time Between Resolution
Thursday 16 June 2011
Technical Debt
Thursday 16 June 2011
Leaks
Thursday 16 June 2011
Debt Metaphornot all debt is created equal
http://www.flickr.com/photos/shoot-art/4536509717/
Thursday 16 June 2011
Customer Responsiveness
Jim HighsmithThursday 16 June 2011
Types of Tech debt
http://forums.construx.com/blogs/stevemcc/archive/2007/11/01/technical-debt-2.aspx
“oops” “yeah I know”
“we’ll fix it the next minor revision”
“we’ll fix it the next major revision”
unintentional intentional
short term (tactical) long term (strategical)
Thursday 16 June 2011
Gartner
• http://www.castsoftware.com/resources/materials/analyst/Gartner-How_to_Monetize_Application_Technical_Debt.pdf
Thursday 16 June 2011
The Vicious Cycle of Technical Debt
1
(More) Relentless Pressure
Take Technical
Debt
Fail to Pay Debt Back
Neglect Maintenance
Technical Debt
Accrues
Diminished Dev Velocity
Vicious Cycle of Technical Debt
Israel Gathttp://theagileexecutive.com/2010/09/20/how-to-break-the-vicious-cycle-of-technical-debt/
Thursday 16 June 2011
Symptoms - Measurable
• Poor Customer Responsiveness
• Long Delivery Times
• Late deliveries
• Lots of defects (Toxic Code)
• Rising Development costs
• Frustrated & Poor performing Teams
http://www.theagileengineer.com/public/Home/Home_files/TechnicalDebt_published.pdf
Thursday 16 June 2011
Secret Sauce
http://radar.oreilly.com/archives/2007/10/Ops-ROI-jesserobbins.html
Thursday 16 June 2011
The Team Knows
http://kingsinsight.com/2010/07/31/ask-the-team-about-technical-debt-the-creeping-doom-graph/
Thursday 16 June 2011
http://farm4.static.flickr.com/3167/2769773425_16c235dc28.jpg
Failure Demand
Value Demand%
As a TeamThursday 16 June 2011
Level 5 - End User LevelThursday 16 June 2011
End User Perspective
• END to END monitoring
• User Experience (browser testing)
• External parties (Email, ISP, Plugins)
Thursday 16 June 2011
Participatory Design
http://www.informaworld.com/smpp/content~db=all~content=a790786652
Thursday 16 June 2011
“Crowd” - MonitoringSudden
Change in UserActivity Pattern
Detection of problem?
A/B Testing
Thursday 16 June 2011
http://www.informaworld.com/smpp/content~db=all~content=a790786652
Spend more time creatively , early in
the processAutomation =>
Traditional Engineering
Thursday 16 June 2011
Design Strategies
http://www.psych.lancs.ac.uk/people/lindenball.html
http://www.amazon.com/Research-Design-Thinking-Nigel-Cross/dp/9062757960 p 4
To Design is Human
To Design does not require super powers
Design used to be a collective or shared ability
Thursday 16 June 2011
Open Innovation
http://en.wikipedia.org/wiki/Open_innovation
http://www.clickadvisor.com/downloads/Bughin_Next_Step_in_Open_Innovation.pdf
Wikipedia
SharedInfrastructures
SharedConfig Management
Open Source
Thursday 16 June 2011
Novice vs Expert
• depth first
• vertical oriented
• can not deal with uncertainty
http://www.psych.lancs.ac.uk/people/lindenball.html
http://www.amazon.com/Research-Design-Thinking-Nigel-Cross/dp/9062757960 p 147
Linden Ball
• breath first , depth dives
• horizontal oriented
• uncertainty = options
Thursday 16 June 2011
Continuous Evolution
“Problem space evolves with the
solution”
http://www.psych.lancs.ac.uk/people/lindenball.html
http://www.amazon.com/Research-Design-Thinking-Nigel-Cross/dp/9062757960
Thursday 16 June 2011
Cultural Taxonomy
Thursday 16 June 2011
Cultural Taxonomy
Thursday 16 June 2011
tl;dr
• avoid vanity metrics
• insight > data
• don`t hate the player
• hate the game
• business value good
• beware of technical debt
Thursday 16 June 2011
Questions?
Thursday 16 June 2011
Thursday 16 June 2011
Thank you!
Thursday 16 June 2011