Yahoo Release Management
-
Upload
phil-hollenback -
Category
Technology
-
view
837 -
download
0
description
Transcript of Yahoo Release Management
![Page 2: Yahoo Release Management](https://reader034.fdocuments.us/reader034/viewer/2022052507/5583ccb8d8b42a784f8b5677/html5/thumbnails/2.jpg)
Hi There
Yahoo! Mailmaybe you’ve heard of it?
Release Managementwe install the softwaressupported by many others in Service Engineering
![Page 3: Yahoo Release Management](https://reader034.fdocuments.us/reader034/viewer/2022052507/5583ccb8d8b42a784f8b5677/html5/thumbnails/3.jpg)
Environment
10k freebsd machinesOther teams handle the hardwareWe focus on just the software installs
![Page 4: Yahoo Release Management](https://reader034.fdocuments.us/reader034/viewer/2022052507/5583ccb8d8b42a784f8b5677/html5/thumbnails/4.jpg)
Houston?
Core mail backend servers means lots of legacy codeAbout 1500 software packages
Mail must be 100% reliablePeople expect email to always be available
It’s hard to load test mail service
![Page 5: Yahoo Release Management](https://reader034.fdocuments.us/reader034/viewer/2022052507/5583ccb8d8b42a784f8b5677/html5/thumbnails/5.jpg)
Solution
Distributed, cached software repositories with distKeep system state in IgorAssemble releases with SRMMassively parallel pushes with PogoRigorous release cycleDO IT FAST
(3 week release timeline)
![Page 6: Yahoo Release Management](https://reader034.fdocuments.us/reader034/viewer/2022052507/5583ccb8d8b42a784f8b5677/html5/thumbnails/6.jpg)
Release Philosophy
All Packages All EnvironmentsPackages must be installed on all servers of a given typeOnly way we can properly test releasesAny other solution leads to many separate ’releases’
![Page 7: Yahoo Release Management](https://reader034.fdocuments.us/reader034/viewer/2022052507/5583ccb8d8b42a784f8b5677/html5/thumbnails/7.jpg)
Which brings us to. . .
![Page 8: Yahoo Release Management](https://reader034.fdocuments.us/reader034/viewer/2022052507/5583ccb8d8b42a784f8b5677/html5/thumbnails/8.jpg)
![Page 9: Yahoo Release Management](https://reader034.fdocuments.us/reader034/viewer/2022052507/5583ccb8d8b42a784f8b5677/html5/thumbnails/9.jpg)
Release Trains
Start a new train every 3 weeksRelease Management and QA hand release back and forthRelease assembly with SRMQA does integrationQA final test & signoffPush to Prod in about a week (with Pogo)
That’s 10k hosts worldwide, folks
![Page 10: Yahoo Release Management](https://reader034.fdocuments.us/reader034/viewer/2022052507/5583ccb8d8b42a784f8b5677/html5/thumbnails/10.jpg)
Delays?
Train delays suckSee the part about DOING IT FAST
too much backup means we cancelroll changes into next train
QA is limiting factor
![Page 11: Yahoo Release Management](https://reader034.fdocuments.us/reader034/viewer/2022052507/5583ccb8d8b42a784f8b5677/html5/thumbnails/11.jpg)
Massively Parallel Pushes
Launch software push on lots of hosts at oncebut with constraints & healthcheck
Starting a push to about 3000 hosts
$ pogo run -R Train_34 -I @mail.farm.xset.deploy-stage4-usPassword:p0000031421; http://pogo.corp.yahoo.com/p0000031421status: running: constraints computed
![Page 12: Yahoo Release Management](https://reader034.fdocuments.us/reader034/viewer/2022052507/5583ccb8d8b42a784f8b5677/html5/thumbnails/12.jpg)
![Page 13: Yahoo Release Management](https://reader034.fdocuments.us/reader034/viewer/2022052507/5583ccb8d8b42a784f8b5677/html5/thumbnails/13.jpg)
Job Complete
10 hours to push to just over 3000 servers:
End of Push
CMR completed successfullyStart Wed Apr 13 2011 22:24:45 GMT-0700 (PDT)End Thu Apr 14 2011 08:24:44 GMT-0700 (PDT) (+10h0s)Total Hosts : 3046 (Skipped 36 invalid hosts)Finished : 3038Completion % : 99.73%
ps - open source pogo can be found athttps://github.com/nrh/pogo/
![Page 14: Yahoo Release Management](https://reader034.fdocuments.us/reader034/viewer/2022052507/5583ccb8d8b42a784f8b5677/html5/thumbnails/14.jpg)
Yes, We Do Rollbacks
Rigorous Release Testingforward and backwardbare metal
Full rollback roughly every 6 months
![Page 15: Yahoo Release Management](https://reader034.fdocuments.us/reader034/viewer/2022052507/5583ccb8d8b42a784f8b5677/html5/thumbnails/15.jpg)
What Works
massively parallel pushesrollbackrigorous release testing
![Page 16: Yahoo Release Management](https://reader034.fdocuments.us/reader034/viewer/2022052507/5583ccb8d8b42a784f8b5677/html5/thumbnails/16.jpg)
Challenges
getting developers to care about old codepackage quality, install/remove scriptsseparating bugfix and feature trainsmoving settings to central config servers
![Page 17: Yahoo Release Management](https://reader034.fdocuments.us/reader034/viewer/2022052507/5583ccb8d8b42a784f8b5677/html5/thumbnails/17.jpg)
To Sum It Up
DON’T BE CLEVER!
Install the same packages everywhereTest releases thoroughly before rolloutThe simple approach is most likely to succeed
![Page 18: Yahoo Release Management](https://reader034.fdocuments.us/reader034/viewer/2022052507/5583ccb8d8b42a784f8b5677/html5/thumbnails/18.jpg)
Thanks for Listening!
Phil [email protected]
@philiphhttp://www.hollenback.net
This Is A Team EffortJen DraperJerrod KensilBrian McNeffJoon Kim
Shajeeb MuhammadPrem RamnathTisha EmmanuelPradipta Ghosh
![Page 19: Yahoo Release Management](https://reader034.fdocuments.us/reader034/viewer/2022052507/5583ccb8d8b42a784f8b5677/html5/thumbnails/19.jpg)
Any Questions?