From vagrant to production - Mark Eijsermans
-
Upload
devopsdays -
Category
Technology
-
view
689 -
download
0
Transcript of From vagrant to production - Mark Eijsermans
![Page 1: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/1.jpg)
From Vagrant to production Mark Eijsermans HootSuite - So"ware Engineer
@markeijsermans http://code.hootsuite.com
![Page 2: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/2.jpg)
• Social media dashboard • 8M users • 40 people committing code • 4 ops • AWS (2/3) & private cloud (1/3) • 100M requests/day (hootsuite.com) • 70M requests/day (ow.ly)
HootSuite
![Page 3: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/3.jpg)
• Monolithic web app (PHP)
• Transitioning towards service oriented architecture (Scala)
![Page 4: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/4.jpg)
In the beginning
dev server (LAMP)
production (LAMP)
dev 1 dev 2
svn
smb smb
![Page 5: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/5.jpg)
Release anytime • Small team • Is intimately aware of prod • Low overhead to release
DevOps?
In the beginning
![Page 6: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/6.jpg)
..a while later
dev server svn
dev dev dev dev dev
dev dev dev dev dev
dev dev dev dev dev
dev dev dev dev dev
dev dev dev dev dev
web web web web
web web web web
web web web web
gearman gearman
0mq worker
cron cron cron cron
smb 0mq worker
ops ops
![Page 7: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/7.jpg)
1 release / 2-4 weeks • Medium sized team • Branching, pre-release code freezes • Only some devs knowledgeable of prod • Ops mostly handles deploy • Complicated process
Devs vs. Ops?
..a while later
![Page 8: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/8.jpg)
…and a few years more (now)
Vagrant
automate
build
test
QA
dev dev dev dev dev
dev dev dev dev dev
dev dev dev dev dev
dev dev dev dev dev
dev dev dev dev dev
dev dev ops ops ops
monolithic web app
Service A Service B
Service C Service D
![Page 9: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/9.jpg)
6 - 10 release / day • Larger team • No branching • Anyone deploys • Automated process • Commit to production in 15min • Much higher % of devs understand prod
…and a few years more (now)
![Page 10: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/10.jpg)
![Page 11: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/11.jpg)
• You’re never going to get a perfect system • How far along are we? • How many things are actually: • Automated • Reproducible • In a state we want them to be
• When can we start feeling good about it?
Perfection is the enemy of good
![Page 12: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/12.jpg)
• svn • svn mirror • PHP • phar • tar + zsync
• Scala • debian packages
The move to artifacts
![Page 13: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/13.jpg)
Dev Build Deploy
Jenkins
project code unit test create artifact
Artifact Repo
build #124
build #123
production
staging
smoke test
production service A/B/C
dash4deploy
![Page 14: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/14.jpg)
![Page 15: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/15.jpg)
![Page 16: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/16.jpg)
![Page 17: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/17.jpg)
![Page 18: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/18.jpg)
![Page 19: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/19.jpg)
![Page 20: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/20.jpg)
![Page 21: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/21.jpg)
“I’m on it”
Broke the build?
![Page 22: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/22.jpg)
![Page 23: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/23.jpg)
• Working on a shared web server over samba was simply painful • extremely volatile environment
• Vagrant simply kicks ass • Dev / prod parity • Encourages experimentation • go ahead and try out that new X
Vagrant
![Page 24: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/24.jpg)
Dev Build Deploy
Jenkins
project code (vagrant)
unit test create artifact
Artifact Repo
build #124
build #123
production
staging
smoke test
production service A/B/C
dash4deploy
![Page 25: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/25.jpg)
Dev Build Deploy
Jenkins
project code (vagrant)
unit test create artifact
Artifact Repo
vagrant.box
build #124
production
staging
smoke test
production service A/B/C
dash4deploy
Ansible build vagrant box
build #123
![Page 26: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/26.jpg)
![Page 27: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/27.jpg)
![Page 28: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/28.jpg)
…yes, use any CM, but why we chose Ansible:
• New team members up to speed in a few days: • Gentle learning curve – YAML
• Push over ssh - easy to understand
• The more declarative it is, the more it documents
• It's all about disseminating information
• Agentless model suits immutable ephemeral instances
Ansible (and why we love it)
![Page 29: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/29.jpg)
Digital archeology
![Page 30: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/30.jpg)
There are gems in that dirt
• logs • you mean logs aren’t saved to /dev/null ? • app was rotating logs, vs. using logrotate
• memcached, mongodb servers not all the same version • what the hell is server X for ???
Auditing our infrastructure
![Page 31: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/31.jpg)
• With configuration management gold images become a caching layer to the build process
• Private cloud - idle instances are cheap
• AMI - need auto spinning to save $$$ & time
Cache for gold
![Page 32: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/32.jpg)
• 200+ server types is hard to manage
• Amortize the stack
• Custom bespoke vs. generic appliances
Simplify for sanity
![Page 33: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/33.jpg)
• Learning happens in production
• Production is still running at 3am
Devs on call
![Page 34: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/34.jpg)
“People who are really serious about so"ware should make their own hardware”
Alan Kay
![Page 35: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/35.jpg)
• Is your monitoring and tooling truly effective?
• Make sure they can ssh into the damn box! (if they have to)
• Throwing some poor dev onto a broken system they know nothing about doesn't work
• Choose your own adventure cookbooks
• Create registry of specialists
Devs on call
![Page 36: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/36.jpg)
Is this working?
![Page 37: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/37.jpg)
• Yammer (& HipChat for on call) • Tried IRC, was too silo’ed
• 5 whys post mortems
• Accountability • Build fails - “I'm on it”
Working out loud
![Page 38: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/38.jpg)
Learning happens in production
![Page 39: From vagrant to production - Mark Eijsermans](https://reader035.fdocuments.us/reader035/viewer/2022062418/554bcbf0b4c905ac708b45c9/html5/thumbnails/39.jpg)
Thank you! @markeijsermans http://code.hootsuite.com