Beyond CD - JAXLondon · @michieltcs. REDUCE TOIL. the kind of work tied to running a production...
107
BEYOND CD LEARN, ADAPT & IMPROVE MICHIEL ROOK @MICHIELTCS
Transcript of Beyond CD - JAXLondon · @michieltcs. REDUCE TOIL. the kind of work tied to running a production...
BEYOND CD LEARN, ADAPT & IMPROVEMICHIEL ROOK@MICHIELTCS
BACKGROUND
@michieltcs
LARGE SEMI-GOVIN NL
@michieltcs
MYSQL ELASTIC
INTERNET
@michieltcs
TOPIC TEAMS
@michieltcs
ON-CALL
@michieltcs
FIRE TEAM
@michieltcs
PACKAGE/DEP UPDATES
@michieltcs
BUG TRIAGE
@michieltcs
OFFICE HOURS INCIDENT RESPONSE
@michieltcs
RELEASE ROLLOUT
@michieltcs
A TYPICAL RELEASE...
@michieltcs
RELEASE CHECKLIST
@michieltcs@michieltcs
Create release
Test release
Deploy release
every2 weeks
Thursday(sprint end)
Monday (sprint start)
Tuesday
@michieltcs
1. BRANCH RC 2. DEPLOY TO ACC 3. SIGNAL TESTER(S) 4. SIGNAL PO 5. FIX ON BRANCH 6. GET APPROVAL 7. DEPLOY TO PROD 8. VERIFY PROD
@michieltcs
2-3 DAYSMANUAL WORK
@michieltcs
... HOT FIXES?
@michieltcs
GOALS
@michieltcs
REDUCE COST
@michieltcs
FAST FEEDBACK
@michieltcs
REDUCE TOIL
@michieltcs
REDUCE TOILthe kind of work tied to running a production service that
tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a
service grows
@michieltcs
PUT PO/BIZ BACK IN DRIVER'S SEAT
@michieltcs
RESPONSIVE ORGANIZATION
@michieltcs
WHY CD THEN?
@michieltcs
REDUCE RISK
@michieltcs
REDUCE WASTE
@michieltcs
REDUCE COST
@michieltcs
IF IT HURTSDO IT MORE OFTEN
@michieltcs
BIG STEPS
@michieltcs
FAIL BIG
@michieltcs
SMALL STEPS
@michieltcs
FAIL SMALL
@michieltcs
$ = REALIZED VALUE
CREDITS TO @FGOULDING
@michieltcs@michieltcs
HIGH VS. LOW PERFORMING TEAMS
Source: Accelerate - The Science Of DevOps
APPROACH
@michieltcs
3 PHASES
@michieltcs
DEFER SPLITTING THE MONOLITH
@michieltcs
INCREASE TEMPO & AUTOMATION
@michieltcs
SINGLE PIPELINE
@michieltcs
IMPROVEMENTS & TESTING
PHASE 1: INCREASING
RELEASE CADENCE
@michieltcs
WEEKLY RELEASE
@michieltcs
AUTOMATION
@michieltcs@michieltcs
Build / Package
AutomatedTests
AcceptanceEnvironment
ProductionEnvironment
@michieltcs@michieltcs
Build / Package
AutomatedTests
AcceptanceEnvironment
ProductionEnvironment
@michieltcs
pipeline
{
stages
{
stag
e('Run t
ests') {
sh
"gradle
check"
}
stage('B
uild doc
ker imag
e') {
sh
"docker
build -
t jobser
vice:${e
nv.BUILD
_NUMBER}
."
sh
"docker
push jo
bservice
:${env.B
UILD_NUM
BER}"
}
stage('D
eploy st
aging')
{
sh
"ansibl
e-playbo
ok -e BU
ILD=${en
v.BUILD_
NUMBER}
-i stagi
ng deplo
y.yml"
}
stage('D
eploy pr
oduction
') {
sh
"ansibl
e-playbo
ok -e BU
ILD=${en
v.BUILD_
NUMBER}
-i prod
deploy.y
ml"
}
}
}
PIPELINE AS CODE
image: registry.local/runner:latest
stages: - test - images - deploy
phpunit: stage: test script: - bin/phpunit
images: stage: images
script: - make images publish
only: - master deploy: stage: deploy
script: - make deploy
only: - master
@michieltcs
WAY OF WORKING
@michieltcs
PAIRPROGRAMMING
BOY SCOUT RULE
@michieltcs
NO MORE RELEASE BRANCHES
PHASE 2: SINGLE PIPELINE
@michieltcs@michieltcs
Build / Package
AutomatedTests
AcceptanceEnvironment
ProductionEnvironment
@michieltcs
CONTINUOUS DEPLOYMENT
@michieltcs
ROLLING DEPLOYS
@michieltcs@michieltcs
LOAD BALANCER
ROLLING UPDATE
SERVICE V1.0 SERVICE V1.0SERVICE V1.0
@michieltcs
@michieltcs@michieltcs
LOAD BALANCER
ROLLING UPDATE
SERVICE V1.0 SERVICE V1.0SERVICE V1.0
SERVICE V1.1
@michieltcs
@michieltcs@michieltcs
LOAD BALANCER
ROLLING UPDATE
SERVICE V1.0 SERVICE V1.1SERVICE V1.0
SERVICE V1.0
@michieltcs
@michieltcs@michieltcs
LOAD BALANCER
ROLLING UPDATE
SERVICE V1.0 SERVICE V1.1SERVICE V1.0
SERVICE V1.1
@michieltcs
@michieltcs@michieltcs
LOAD BALANCER
ROLLING UPDATE
SERVICE V1.0 SERVICE V1.1SERVICE V1.1
SERVICE V1.0
@michieltcs
@michieltcs@michieltcs
LOAD BALANCER
ROLLING UPDATE
SERVICE V1.0 SERVICE V1.1SERVICE V1.1
SERVICE V1.1
@michieltcs
@michieltcs@michieltcs
LOAD BALANCER
ROLLING UPDATE
SERVICE V1.1 SERVICE V1.1SERVICE V1.1
SERVICE V1.0
@michieltcs
@michieltcs@michieltcs
LOAD BALANCER
ROLLING UPDATE
SERVICE V1.1 SERVICE V1.1SERVICE V1.1
@michieltcs
@michieltcs@michieltcs
@michieltcs@michieltcs
@michieltcs@michieltcs
@michieltcs
PIPELINE FAILURES
@michieltcs
FLAKY TESTS
@michieltcs
TIMEOUTS
@michieltcs
EXTERNAL DEPS IN TESTS
@michieltcs
NETWORK STABILITY ISSUES
@michieltcs
BRITTLE INFRASTRUCTURE
@michieltcs
IF YOU CAN'T TRUST IT - REMOVE IT
@michieltcs
EXTREME FEEDBACK
STOP THE LINE
@michieltcs@michieltcs
PHASE 3: IMPROVEMENTS &
TESTING
@michieltcs
PIPELINE SPEED
@michieltcs
BUY A BIGGER BOX
@michieltcs
CLEVER PIPELINE IMPROVEMENTS:
PARALLELIZE
@michieltcs
TESTING IMPROVEMENTS
@michieltcs@michieltcs
UNITTESTS
E2E /VISUAL TESTS
INTEGRATIONTESTS
LOTS OF MANUAL TESTING
E2E TESTS
@michieltcs
AUTOMATION
@michieltcs@michieltcs
CONTRACT TESTING
@michieltcs@michieltcs
UI TESTING
@michieltcs@michieltcs
VISUAL TESTING
@michieltcs@michieltcs
UNIT TESTS
UI /VISUAL TESTS
Exploratorytesting & user
feedback
Monitoring& alerting
INTEGRATION / CONTRACTTESTS
COST SPEED
@michieltcs
RELEASEAPPROVAL?
FEATURE TOGGLES
@michieltcs
DECOUPLEDEPLOYMENTS
FROMRELEASES
@michieltcs@michieltcs
@michieltcs@michieltcs
@michieltcs
CAUTION:FEATURE TOGGLE
DEBT
RESULTS & TAKEAWAYS
@michieltcs
LEARNINGS
@michieltcs
PIPELINE SPEED & STABILITY
@michieltcs
WAY OF WORKING & MINDSET
@michieltcs
VISUAL TESTING TOOLS
@michieltcs
(AUTOMATED) SECURITY TESTING
@michieltcs
SOME STATS
@michieltcs
~4000 TESTS PER DEPLOYMENT
@michieltcs
~10 MINUTES PERPIPELINE RUN
@michieltcs
~1.5 HOURS FROM DEV TO PROD
@michieltcs
IN 6 MONTHS:MORE THAN 5000
DEPLOYMENTS
