Dev ops with smell v1.2

DEVOPS ANTIPATTERNS

~WHOAMI

ANTONS KRANGA▸ Full stack developer ~ 15years

▸ Cloud Architect

▸ DevOps Advocate

▸ Full-Stack Developer

▸ Speaker

▸ Marathon runner@acankr

TALK STRUCTURE

PRESO PLAN

▸ Smell Symptoms: organizational anti patterns for devops

▸ Vitamins: proposed solutions from devops library

▸ Painkillers: deployment anti patterns

▸ Antibiotics: environment anti patterns brought by devops

pic from wonderful “matrix” movie

WHY DEVOPSDEVELOPERS VERSION

DEFINITION OF DEVOPS

What is DevOps?

pic from “hitchhiker's guide to the galaxy” movie

What is DevOps?

BFG900 from classical game DOOM (c) id software

What is DevOps?

▸ Development teams ▸ Operations

▸ Security and compliance ▸ Management

▸ Strategy, IT Strategy

BFG900 from classical game DOOM (c) id software

▸ QA▸ HR

DRIVERS FOR SOFTWARE DEVELOPMENT

▸ development price

BEFORE AFTER

▸ innovations speed

DRIVERS FOR SOFTWARE DEVELOPMENT

▸ development price

BEFORE AFTER

▸ innovations speed

▸ specialization silos

▸ ops comes first

▸ cross functional

▸ software defined data centers

▸ complex communication ▸ complexity theory

SMELL SYMPTOMDEVOPS ANTIPATTERNS

ANTIPATTERN # 0

BI MODAL IT

Sparky from Frankenweenie

▸ Slow IT

▸ Safe and Predictable

▸ IT Centric

▸ Secure and Regulated

▸ Fast IT

▸ Fast and Agile

▸ Business Centric

▸ Innovative

ANTIPATTERN # 1

DEVOPS IN A BOX

You cannot buy culture!

▸ Buy a Golden DevOps LVL5

▸ Become certified DevOps master

▸ Give me DevOps compliance checklist

▸ Give me 5 key DevOps control metrics

ANTIPATTERN # 1.1

REBRANDED IT

DevOps != Configuration ManagementDevOps != Release Management|DevOps != Product Management…DevOps != (*) Management

DEVOPS IS THE CULTURE YOU CANNOT FIND IN IN ITIL CATALOG

ANTIPATTERN # 2

DEVOPS AS SEPARATE TEAM

You don’t need to change anything in your org

You extend organization with new capabilities

ANTIPATTERN # 2

DEVOPS AS SEPARATE TEAM

Creates Unicorn vs Horses dilemma

DevOps builds their own Silo

Often loses focus on value and speed

ANTIPATTERN # 2.1

DEVOPS AS A SILO

DevOps teams build their own Silo

▸ You cannot talk to DevOps

▸ Use Jira instead !!!!

▸ RTFM Architecture

ANTIPATTERN # 2: PROPOSED SOLUTION

SORRY, NOT MY DEPARTMENT

Bread ownership and specialization with autonomous teams

http://martinfowler.com/bliki/DevOpsCulture.html

ANTIPATTERN # 3

DEVOPS IN AGILE

▸ DoD fails with “ilities”

▸ Services can be easily “undone”

ANTIPATTERN # 4

FEAR OF RELEASE

RELEASE === RISK

NO RELEAE NO REVENUE

ANTIPATTERN # 5

SNOWFLAKE SERVER

Applying changes to Server Instance manually leads to unique and distinct server configuration footprint (TECHNICAL DEBT)

http://martinfowler.com/bliki/SnowflakeServer.html

VITAMINSDEVOPS ANTIPATTERNS

VITAMINS

INFRASTRUCTURE AS CODE

DEV …CODE

VITAMINS

Code is aContract OPS

Essence to collectively create and share knowledge

VITAMINS

Enables: built in quality

VITAMINS

Put infrastructure into SCM

VITAMINS

Code is aContract OPS`

Make infrastructure part of app

VITAMINS

USE CI ENVIRONMENT

test infra depl AT

...release

VITAMINS

USE CI ENVIRONMENT

▸ We need feedback not to be afraid

▸ Feedback != SPAM

▸ It’s about size of release not frequency

▸ Don’t judge for broken builds

▸ Go home when build is green

PAINKILLERSDEVOPS ANTIPATTERNS

CONTINUOUS DELIVERY

CI/CD PROCESS

test infra depl AT

...release

ANTIPATTERN # 6

SLOW PIPELINES

...release

test infra depl DT

test DT depl infra

infra depl DT AT

SLOW PIPELINES

Design for parallelization

ANTIPATTERN # 6

“JENKINS” DRIVEN DEVELOPMENT

With single master CI you easily get a single point of failure

CI master

MULTI MASTER “JENKINS”

Multi-master CI can dedicate CI Master per group of related components

CI master CI master CI master

ANTIPATTERN # 6: PROPOSED SOLUTION # 2

SHARE RESPONSIBILITY

CI users (DEVs or OPs) are best for managing their jobs

▸ Implement Pipeline as Code

▸ Store CI/CD pipelines in git

▸ Let pipeline evolve together with your app

ANTIPATTERN # 7

MANUAL PROMOTION

Tester PRODUATIntegr TestsDEV System Tests

ENVENVENV ENV

KILL SWITCH FOR MANUAL TESTING

PRODUATIntegr TestsDEV System Tests

ENVENVENV ENV

KILL SWITCH FOR MANUAL TESTING

PRODUATIntegr TestsDEV System Tests

ENVENVENV ENV

Time adjustableif TREND is good

AB TESTING

PROD A

Integr TestsDEV System Tests

ENVENV

TEST A

PROD B

TEST Bfeedback

feedback

measure

ANTIPATTERN # 7.1

CODEREVIEW

PRODUATIntegr Tests System Tests

ENVENVENV ENV

feature branch DEV Ccode

review

ANTIPATTERN # 7.1: PROPOSED SOLUTION

REACTIVE CODEREVIEW

ReleaseIntegr Tests System Tests

ENVENV ENV

Full Regression

Calc technicaldebt

Nightly

Standup

code review

ANTIPATTERN # 7.1

CODEREVIEW

PRODUATIntegr TestsDEV A System Tests

ENVENVENV ENVmaster

feature branch DEV Ccode

review

ANTIPATTERN # 7.2

GIT FLOWS

▸ Git flow helps to work feature dev isolated

▸ Indirectly breaks Continuous Delivery Principle

▸ Complex merge routine distracts teams

http://classicprogrammerpaintings.com/post/142586036029/junior-programmer-learns-git-rebase

GIT FLOWS

develop

master

GIT FLOWS

develop

master

DEVpull

ANTIPATTERN # 8

DEV DOESN’T CONTROL DEV ENV

DEVENV

…DEV CI

DEV …NO CI

DEVENV

…DEV NO CI

WANT TO HACK? SURE!

DEVENV

YOU BROKE IT? YOU FIX IT!

DEV NO CI

ANTIBIOTICSDEVOPS ANTIPATTERNS

ANTIPATTERN # 9

GOLDEN IMAGE

Problems

▸ Maintained manually

▸ No collaboration

▸ Hard to distribute

▸ Non versioning

Chnorr Service

PROVISIONING

OS Chnorr Service

PROVISIONING

Configure

Infrastructurecode

Harden Download Install

Chnorr Service

PROVISIONING

Configure

Infrastructurecode

Chnorr Service

PROVISIONING

Configure

Infrastructurecode

Chnorr Service

ANTIPATTERN # A

FRAGILE PROVISIONING

Configure

Infrastructurecode

▸ Provisioning success < 100%

▸ Time to provision

Chnorr Service

ANTIPATTERN # A

Configure

Infrastructurecode

Chnorr Service

ANTIPATTERN # A

Configure

Infrastructurecode

Chnorr Service

ANTIPATTERN # A

Configure

Infrastructurecode

Chnorr Service

ANTIPATTERN # A.1

PRIVATE DATA CENTER PROVISIONING

PRIVATE DATA CENTER: NO INTERNET!!!

ConfigureHarden Download Install

Chnorr Service

ANTIPATTERN # A.1

apt yum gem

Chnorr Service

ANTIPATTERN # A.1

apt yum gem

complicated?

Chnorr Service

ANTIPATTERN # A.1: PROPOSED SOLUTION

STEM CELLS

static dynamic

Chnorr Service

STEM CELLS

LAB PRIVATE DATA CENTER

Chnorr Service

STEM CELLS

LAB PRIVATE DATA CENTER

packerPROVISIONMAKE

SNAPSHOT

docker

Chnorr Service

RECONFIGURE

ANTIPATTERN # B

DEPENDENCY HELL

Configure

Infrastructurecode

▸ Version of libraries

▸ Version of packages

▸ Maintain dependencies

Chnorr Service

ANTIPATTERN # B

DEPENDENCY HELL

Configure

Infrastructurecode

▸ Version of libraries

▸ Version of packages

▸ Version of your infra code

Chnorr Service

ANTIPATTERN # B: PROPOSED SOLUTION

CANARY BUILDS

Accept TestsCI

Unit Test Provision

PRECISE version libs

Accept TestsCI Unit Test Provision

LATEST version libs

Chnorr Service

ANTIPATTERN # C

INFRASTRUCTURE PETAttributes of Pet

▸ Have meaningful names

▸ Long living instance

▸ Often needs manual nursing

▸ Requires scary patching

▸ Leads to snowflakes

▸ PaaS is modern pet

ANTIPATTERN # C: PROPOSED SOLUTION

INFRASTRUCTURE CATTLEAttributes of Pet▸ Have numbers in its name

▸ Short living instance

▸ Immutable configuration

▸ Recreate instead of patching

▸ Requires careful planning

PET TO TRANSFORMATION CATTLE EXAMPLE

Chnorr Service

Database

Chnorr Service

Database

Chnorr Service

Database

Cattle

Chnorr Service

Database

Cattle

User Data

ANTIPATTERN # D

SECRETS LEAK

Configure

Infrastructurecode

Install

cmdbwrong place

for your secretswrong place

for your secrets

Chnorr Service

ANTIPATTERN # D: PROPOSED SOLUTION

SECRETS LEAK

▸ Don’t store secrets with code

▸ Don’t store secrets with configuration

▸ Don’t leave secrets in service

Secrets DON’Ts

SECRETS LEAK

▸ Secret can be leased and rotated

▸ tmpfs is your fiend

Secrets DOs

SECURITY LEASING EXAMPLE

Chnorr Service vault

consul

Database

x hours leasing

TAKEAWAYSGOOD INFRA CODE

TAKEAWAYS

PATTERNS GOES BEFORE TOOLS

Patterns Tools

TAKEAWAYS

EVERYTHING MUST HAVE AN API

TAKEAWAYS

SPLIT YOUR SECRETS FROM CODE OR CONFIGURATION

(c) @eduardsi

TAKEAWAYS

SELF TESTABLE CODE

▸ Use assertions for infrastructure code

▸ Use acceptance test frameworks

Tests improves your confidence

CHEF MINITEST

package"mysql"

assert_file"/etc/my.cnf","mysql","mysql","644"

apt-getinstall-yqqmysql

test!-f/etc/my.cnf||error_exit“boo”

TAKEAWAYS

HOW TO START DEVOPS

▸ Fastest way to start with Vitamins and Coaching

▸ Apply measures early

▸ Apply right measures

▸ Tools comes later

TAKEAWAYS

READING

▸ Book: The Phoenix Project

▸ Author: Gene Kim Kevin Behr George Stafford

▸ ISBN: 978-0988262508

TAKEAWAYS

READING

▸ Book: The Lean Startup

▸ Author: Eric Ries

▸ ISBN: 978-0307887894

TAKEAWAYS

READING

▸ Book: A Human Error Approach to Aviation Accident Analysis

▸ Author: Douglas A. Wiegmann Scott A. Shappell

▸ ISBN: 978-0754618737

THANK YOU

Download me here: http://www.slideshare.net/akranga/dev-ops-with-smell-v12

Twitter: @acankr

Dev ops with smell v1.2

Software

Transcript of Dev ops with smell v1.2

AstrixRadioPlanning v1.2

Dev Ops without the Ops

Smell technology

Thebigpicture v1.2

Smell “test”!

report v1.2

Equipment v1.2

Sigh, smell touch, see, tuh, uh smell

Tutorial v1.2

COGNOS_Development_Guidelines_and_Standards V1.2

Code Smell

Smell (Olfaction). Smell is a chemical sense. You inhale something of whatever of whoever it is you smell. You smell something when molecules of a substance.

SMELL DISORDERS

Scalemail V1.2

Senses smell

Smell Smell and Taste - University of Sussex · 2003-11-13 · 1 Smell and Taste Perception and Attention Course George Mather Smell • Smell is usually considered a minor sense

To smell or not to smell…

OASIS Open Document Format for Office Applications ...docs.oasis-open.org/office/v1.2/os/OpenDocument-v1.2-os.pdf · OpenDocument v1.2 part 1: OpenDocument Schema OpenDocument v1.2

Bibliomula v1.2

Dev ops != Dev+Ops