The Road to Continuous Deployment
45
Transcript of The Road to Continuous Deployment
THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY
MICHIEL ROOK
@michieltcs
THE SYSTEM - SAN DIEGO
▸ ... or the Big Ball Of Mud
▸ Large legacy monolith
▸ Generates significant income
▸ Slow & complex
▸ Technical debt
@michieltcs
SAN DIEGO FRONTEND
MYSQL DB
SAN DIEGO BACKEND
LOAD BALANCERS / VARNISH
ITBANEN INTERMEDIAIR NATIONALEVACATUREBANK
SAN DIEGO FRONTEND
SAN DIEGO FRONTEND
SAN DIEGO FRONTEND
SAN DIEGO BACKEND
SAN DIEGO BACKEND
SAN DIEGO BACKEND
MEMCACHE FTP EXT. SERVICES
SOLR
@michieltcs
THE SYSTEM - SAN DIEGO
▸ Infrequent, manual releases
▸ Fragile tests
▸ Frequent outages & issues
▸ Frustrated team
▸ Low confidence modifying existing code
@michieltcs
GOALS
▸ Reduce issues
▸ Reduce cycle time
▸ Increase productivity
▸ Increase motivation
REFACTOR? REBUILD?
APPROACH
▸ Strangler pattern
▸ API first
▸ Services per domain object (job, jobseeker, ...)
▸ Migrate individual pages
@michieltcs
ORIGINAL MONOLITH
PROXY
SERVICEORIGINAL MONOLITH
ORIGINAL MONOLITH
SERVICE SERVICE
SERVICE
PROXY
DB
DBDB
DB
DB DB
@michieltcs
APPROACH
▸ Services behind load balancers
▸ Access legacy db’s
▸ Continuous deployment
▸ Docker containers
▸ Frontends are services
@michieltcs
CONTINUOUS EVERYTHING
DEV BUILD / TEST
CONTINUOUS INTEGRATION
@michieltcs
DEV BUILD / TEST ACCEPTANCE PRODUCTION
CONTINUOUS DELIVERY
@michieltcs
DEV BUILD / TEST STAGING PRODUCTION
CONTINUOUS DEPLOYMENT
@michieltcs
WHY CONTINUOUS DEPLOYMENT
▸ Small steps
▸ Early feedback
▸ Reduce cycle time
▸ Reduce risk
▸ Experiments!
@michieltcs
SUCCESSFUL TEAMS HAVE
@michieltcs
ONLY COMMIT TO MASTER
EVERY COMMIT GOES TO PRODUCTION
PAIR PROGRAMMING
QUALITY GATES
FEATURE TOGGLES, A/B TESTS
@michieltcs
DASHBOARDS
DEVOPS
AUTOMATE REPEATABLE THINGS
CONTINUOUS TESTING
CONTINUOUS TESTING
UNIT TESTS
INTEGRATION TESTS
ACCEPTANCE TESTS
UI TESTS
SMOKETESTS
Cost Speed
Exploratorytesting Monitoring
@michieltcs
PIPELINE AS CODE
node { stage 'Run tests' sh "phpunit" sh "behat" stage 'Build docker image' sh "phing build" sh "docker build -t jobservice:${env.BUILD_NUMBER} ." sh "docker push jobservice:${env.BUILD_NUMBER}" stage 'Deploy acceptance' sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER} -i acc deploy.yml" stage 'Deploy production' sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER} -i prod deploy.yml" }
@michieltcs
DEPLOYING
PULL IMAGE
START NEW CONTAINER
WAIT FOR PORT
SMOKE TESTS / HEALTH CHECKS
ADD NEW CONTAINER TO LB
REMOVE OLD CONTAINER FROM LB
STOP OLD CONTAINER
docker pull
@michieltcs
DEPLOYING
PULL IMAGE
START NEW CONTAINER
WAIT FOR PORT
SMOKE TESTS / HEALTH CHECKS
ADD NEW CONTAINER TO LB
REMOVE OLD CONTAINER FROM LB
STOP OLD CONTAINER
docker run
@michieltcs
DEPLOYING
PULL IMAGE
START NEW CONTAINER
WAIT FOR PORT
SMOKE TESTS / HEALTH CHECKS
ADD NEW CONTAINER TO LB
REMOVE OLD CONTAINER FROM LB
STOP OLD CONTAINER
wait_for: port=8080 delay=5 timeout=15
@michieltcs
DEPLOYING
PULL IMAGE
START NEW CONTAINER
WAIT FOR PORT
SMOKE TESTS / HEALTH CHECKS
ADD NEW CONTAINER TO LB
REMOVE OLD CONTAINER FROM LB
STOP OLD CONTAINER
uri: url: http://localhost:8080/_health status_code: 200 timeout: 30
@michieltcs
DEPLOYING
PULL IMAGE
START NEW CONTAINER
WAIT FOR PORT
SMOKE TESTS / HEALTH CHECKS
ADD NEW CONTAINER TO LB
REMOVE OLD CONTAINER FROM LB
STOP OLD CONTAINER
template: src=haproxy.cfg.j2 dest=/etc/haproxy/haproxy.cfg
service: name=haproxy state=reloaded
@michieltcs
DEPLOYING
PULL IMAGE
START NEW CONTAINER
WAIT FOR PORT
SMOKE TESTS / HEALTH CHECKS
ADD NEW CONTAINER TO LB
REMOVE OLD CONTAINER FROM LB
STOP OLD CONTAINER
template: src=haproxy.cfg.j2 dest=/etc/haproxy/haproxy.cfg
service: name=haproxy state=reloaded
@michieltcs
DEPLOYING
PULL IMAGE
START NEW CONTAINER
WAIT FOR PORT
SMOKE TESTS / HEALTH CHECKS
ADD NEW CONTAINER TO LB
REMOVE OLD CONTAINER FROM LB
STOP OLD CONTAINER
docker stop
docker rm
@michieltcs
BUILD PIPELINE
@michieltcs
FEEDBACK!
RESULTS
RESULTS
▸ Total build time per service < 10 minutes
▸ Significantly improved page load times
▸ Improved audience stats (time on page, pages per session, session duration, traffic, seo ranking, etc)
▸ Increased confidence and velocity
▸ Experimented with new tech/stacks (angular, jvm, event sourcing)
▸ More fun
@michieltcs
LESSONS LEARNED
▸ Team acceptance
▸ Change is hard
▸ Mentality / discipline
▸ Docker stability/orchestration
▸ Issues with traffic between Amazon <-> on-premise datacenter
@michieltcs
LESSONS LEARNED
▸ Experience with new tech
▸ Stability of build pipelines
▸ Business alignment
▸ Limit feature toggles
▸ Keep focus on replacing legacy application
@michieltcs
Turn-key Continuous Deployment
Zero downtime deployments
Modern, autoscaling infrastructure with built-in monitoring
Pipeline in five minutes
