Dev ops with smell v1.2

DEVOPS ANTIPATTERNS

~WHOAMI

ANTONS KRANGA▸ Full stack developer ~ 15years

▸ Cloud Architect

▸ DevOps Advocate

▸ Full-Stack Developer

▸ Speaker

▸ Marathon runner@acankr

TALK STRUCTURE

PRESO PLAN

▸ Smell Symptoms: organizational anti patterns for devops

▸ Vitamins: proposed solutions from devops library

▸ Painkillers: deployment anti patterns

▸ Antibiotics: environment anti patterns brought by devops

pic from wonderful “matrix” movie

WHY DEVOPSDEVELOPERS VERSION

DEFINITION OF DEVOPS

What is DevOps?

pic from “hitchhiker's guide to the galaxy” movie


What is DevOps?

BFG900 from classical game DOOM (c) id software


What is DevOps?

▸ Development teams ▸ Operations

▸ Security and compliance ▸ Management

▸ Strategy, IT Strategy

BFG900 from classical game DOOM (c) id software

▸ QA▸ HR

DRIVERS FOR SOFTWARE DEVELOPMENT

▸ development price

BEFORE AFTER

▸ innovations speed

DRIVERS FOR SOFTWARE DEVELOPMENT

▸ development price

BEFORE AFTER

▸ innovations speed

▸ specialization silos

▸ ops comes first

▸ cross functional

▸ software defined data centers

▸ complex communication ▸ complexity theory

SMELL SYMPTOMDEVOPS ANTIPATTERNS

ANTIPATTERN # 0

BI MODAL IT

Sparky from Frankenweenie

▸ Slow IT

▸ Safe and Predictable

▸ IT Centric

▸ Secure and Regulated

▸ Fast IT

▸ Fast and Agile

▸ Business Centric

▸ Innovative

ANTIPATTERN # 1

DEVOPS IN A BOX

You cannot buy culture!

▸ Buy a Golden DevOps LVL5

▸ Become certified DevOps master

▸ Give me DevOps compliance checklist

▸ Give me 5 key DevOps control metrics

ANTIPATTERN # 1.1

REBRANDED IT

DevOps != Configuration ManagementDevOps != Release Management|DevOps != Product Management…DevOps != (*) Management

DEVOPS IS THE CULTURE YOU CANNOT FIND IN IN ITIL CATALOG

ANTIPATTERN # 2

DEVOPS AS SEPARATE TEAM

You don’t need to change anything in your org

You extend organization with new capabilities

ANTIPATTERN # 2

DEVOPS AS SEPARATE TEAM

Creates Unicorn vs Horses dilemma

DevOps builds their own Silo

Often loses focus on value and speed

ANTIPATTERN # 2.1

DEVOPS AS A SILO

DevOps teams build their own Silo

▸ You cannot talk to DevOps

▸ Use Jira instead !!!!

▸ RTFM Architecture

ANTIPATTERN # 2: PROPOSED SOLUTION

SORRY, NOT MY DEPARTMENT

Bread ownership and specialization with autonomous teams

vs

http://martinfowler.com/bliki/DevOpsCulture.html

ANTIPATTERN # 3

DEVOPS IN AGILE

▸ DoD fails with “ilities”

▸ Services can be easily “undone”

ANTIPATTERN # 4

FEAR OF RELEASE

RELEASE === RISK

NO RELEAE NO REVENUE

vs

ANTIPATTERN # 5

SNOWFLAKE SERVER

Applying changes to Server Instance manually leads to unique and distinct server configuration footprint (TECHNICAL DEBT)

http://martinfowler.com/bliki/SnowflakeServer.html

VITAMINSDEVOPS ANTIPATTERNS

VITAMINS

INFRASTRUCTURE AS CODE

ENV

ENV

ENV

DEV …CODE

VITAMINS


DEV

Code is aContract OPS

Essence to collectively create and share knowledge

VITAMINS


DEV


Enables: built in quality

VITAMINS


DEV


Put infrastructure into SCM

VITAMINS


DEV

Code is aContract OPS`

Make infrastructure part of app

VITAMINS

USE CI ENVIRONMENT

CI

test infra depl AT

...release

VITAMINS

USE CI ENVIRONMENT

▸ We need feedback not to be afraid

▸ Feedback != SPAM

▸ It’s about size of release not frequency

▸ Don’t judge for broken builds

▸ Go home when build is green

PAINKILLERSDEVOPS ANTIPATTERNS

CONTINUOUS DELIVERY

CI/CD PROCESS

CI

test infra depl AT

...release

ANTIPATTERN # 6

SLOW PIPELINES

CI

...release

test infra depl DT

test DT depl infra

infra depl DT AT


SLOW PIPELINES

`

`

`

Design for parallelization

ANTIPATTERN # 6

“JENKINS” DRIVEN DEVELOPMENT

With single master CI you easily get a single point of failure

CI master


MULTI MASTER “JENKINS”

Multi-master CI can dedicate CI Master per group of related components

CI master CI master CI master

ANTIPATTERN # 6: PROPOSED SOLUTION # 2

SHARE RESPONSIBILITY

CI users (DEVs or OPs) are best for managing their jobs

▸ Implement Pipeline as Code

▸ Store CI/CD pipelines in git

▸ Let pipeline evolve together with your app

ANTIPATTERN # 7

MANUAL PROMOTION

Tester PRODUATIntegr TestsDEV System Tests

ENVENVENV ENV


KILL SWITCH FOR MANUAL TESTING

PRODUATIntegr TestsDEV System Tests

ENVENVENV ENV

TEST


KILL SWITCH FOR MANUAL TESTING

PRODUATIntegr TestsDEV System Tests

ENVENVENV ENV

TEST

Time adjustableif TREND is good


AB TESTING

PROD A

Integr TestsDEV System Tests

ENV

ENVENV

TEST A

PROD B

ENV

TEST Bfeedback

feedback

measure

ANTIPATTERN # 7.1

CODEREVIEW

PRODUATIntegr Tests System Tests

ENVENVENV ENV

DEV B

feature branch DEV Ccode

review

ANTIPATTERN # 7.1: PROPOSED SOLUTION

REACTIVE CODEREVIEW

ReleaseIntegr Tests System Tests

ENVENV ENV

Full Regression

Calc technicaldebt

ENV

Nightly

DEV A

Standup

code review

…

ANTIPATTERN # 7.1

CODEREVIEW

PRODUATIntegr TestsDEV A System Tests

ENVENVENV ENVmaster

DEV B

feature branch DEV Ccode

review

ANTIPATTERN # 7.2

GIT FLOWS

▸ Git flow helps to work feature dev isolated

▸ Indirectly breaks Continuous Delivery Principle

▸ Complex merge routine distracts teams

http://classicprogrammerpaintings.com/post/142586036029/junior-programmer-learns-git-rebase


GIT FLOWS

develop

master

test

merge

CI

DEV

pull

push


GIT FLOWS

develop

master

test

merge

CI

DEV

DEVpull

push

ANTIPATTERN # 8

DEV DOESN’T CONTROL DEV ENV

DEVENV

DEVENV

DEVENV

…DEV CI



DEV …NO CI

DEVENV

DEVENV

DEVENV



ENV

ENV

DEVENV

…DEV NO CI

WANT TO HACK? SURE!



ENV

ENV

DEVENV

…

YOU BROKE IT? YOU FIX IT!

DEV NO CI

ANTIBIOTICSDEVOPS ANTIPATTERNS

ANTIPATTERN # 9

GOLDEN IMAGE

VM

OS

Problems

▸ Maintained manually

▸ No collaboration

▸ Hard to distribute

▸ Non versioning

Chnorr Service


PROVISIONING

OS Chnorr Service


PROVISIONING

OS

Configure

Infrastructurecode

Harden Download Install

Chnorr Service


PROVISIONING

OS

Configure

Infrastructurecode


cmdb

Chnorr Service

ANTIPATTERN # A

FRAGILE PROVISIONING

OS

Configure

Infrastructurecode


▸ Provisioning success < 100%

▸ Time to provision

Chnorr Service

ANTIPATTERN # A.1

PRIVATE DATA CENTER PROVISIONING

PRIVATE DATA CENTER: NO INTERNET!!!

OS

ConfigureHarden Download Install

Chnorr Service

ANTIPATTERN # A.1



OS


…

apt yum gem

Chnorr Service

ANTIPATTERN # A.1



OS


…

apt yum gem

complicated?

Chnorr Service

ANTIPATTERN # A.1: PROPOSED SOLUTION

STEM CELLS

static dynamic

OS


Chnorr Service


STEM CELLS

OS


LAB PRIVATE DATA CENTER

Chnorr Service


STEM CELLS

OS


LAB PRIVATE DATA CENTER

code

packerPROVISIONMAKE

SNAPSHOT

docker

Chnorr Service

RECONFIGURE

ANTIPATTERN # B

DEPENDENCY HELL

OS

Configure

Infrastructurecode


▸ Version of libraries

▸ Version of packages

▸ Maintain dependencies

Chnorr Service

ANTIPATTERN # B

DEPENDENCY HELL

OS

Configure

Infrastructurecode


▸ Version of libraries

▸ Version of packages


▸ Version of your infra code


Chnorr Service

ANTIPATTERN # B: PROPOSED SOLUTION

CANARY BUILDS

Accept TestsCI

Unit Test Provision

…

PRECISE version libs

Accept TestsCI Unit Test Provision

…

LATEST version libs

Chnorr Service

Chnorr Service

ANTIPATTERN # C

INFRASTRUCTURE PETAttributes of Pet

▸ Have meaningful names

▸ Long living instance

▸ Often needs manual nursing

▸ Requires scary patching

▸ Leads to snowflakes

▸ PaaS is modern pet

ANTIPATTERN # C: PROPOSED SOLUTION

INFRASTRUCTURE CATTLEAttributes of Pet▸ Have numbers in its name

▸ Short living instance

▸ Immutable configuration

▸ Recreate instead of patching

▸ Requires careful planning


PET TO TRANSFORMATION CATTLE EXAMPLE

Chnorr Service

Database



Chnorr Service

Database

Cattle

Pet



Chnorr Service

Database

Cattle

Pet

User Data

ANTIPATTERN # D

SECRETS LEAK

OS

Configure

Infrastructurecode

Install

cmdbwrong place

for your secretswrong place

for your secrets

Chnorr Service

ANTIPATTERN # D: PROPOSED SOLUTION

SECRETS LEAK

▸ Don’t store secrets with code

▸ Don’t store secrets with configuration

▸ Don’t leave secrets in service

Secrets DON’Ts


SECRETS LEAK

▸ Secret can be leased and rotated

▸ tmpfs is your fiend

Secrets DOs


SECURITY LEASING EXAMPLE

Chnorr Service vault

consul

IAM

Database

api

x hours leasing

AWS

TAKEAWAYSGOOD INFRA CODE

TAKEAWAYS

PATTERNS GOES BEFORE TOOLS

Patterns Tools

vs

TAKEAWAYS

EVERYTHING MUST HAVE AN API

REST

DSL

CLI

TAKEAWAYS

SPLIT YOUR SECRETS FROM CODE OR CONFIGURATION

(c) @eduardsi

TAKEAWAYS

SELF TESTABLE CODE

▸ Use assertions for infrastructure code

▸ Use acceptance test frameworks

Tests improves your confidence

CHEF MINITEST

package"mysql"

assert_file"/etc/my.cnf","mysql","mysql","644"

SHELL

apt-getinstall-yqqmysql

test!-f/etc/my.cnf||error_exit“boo”

TAKEAWAYS

HOW TO START DEVOPS

▸ Fastest way to start with Vitamins and Coaching

▸ Apply measures early

▸ Apply right measures

▸ Tools comes later

TAKEAWAYS

READING

▸ Book: The Phoenix Project

▸ Author: Gene Kim Kevin Behr George Stafford

▸ ISBN: 978-0988262508

TAKEAWAYS

READING

▸ Book: The Lean Startup

▸ Author: Eric Ries

▸ ISBN: 978-0307887894

TAKEAWAYS

READING

▸ Book: A Human Error Approach to Aviation Accident Analysis

▸ Author: Douglas A. Wiegmann Scott A. Shappell

▸ ISBN: 978-0754618737

THANK YOU

Download me here: http://www.slideshare.net/akranga/dev-ops-with-smell-v12

Twitter: @acankr

http://www.slideshare.net/akranga/dev-ops-with-smell-v12

Dev ops with smell v1.2

Software

Transcript of Dev ops with smell v1.2