Adaptive Delivery at Scale With the Scaled Agile Framework (SAFe)
Delivery at Scale
-
Upload
adrian-perreau-de-pinninck -
Category
Engineering
-
view
2.002 -
download
1
description
Transcript of Delivery at Scale
![Page 1: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/1.jpg)
Delivery at Scale
![Page 2: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/2.jpg)
Manu Cupcic and Adrian Perreau
![Page 3: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/3.jpg)
Scale
● > 38 PB on our HDFS
● > 1 billion ad impressions per day
● > 5000 servers world wide
![Page 4: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/4.jpg)
cost of change
number of devs
The problem
![Page 5: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/5.jpg)
Once upon a time
![Page 6: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/6.jpg)
Single C# Repository
![Page 7: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/7.jpg)
Build on local machines
![Page 8: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/8.jpg)
Build on local machines
![Page 9: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/9.jpg)
Build on local machines
![Page 10: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/10.jpg)
Continuous Integration??
![Page 11: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/11.jpg)
Running applications
![Page 12: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/12.jpg)
No Testing Culture
![Page 13: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/13.jpg)
Weekly Merges
20 engineers
50 engineers
![Page 14: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/14.jpg)
Weekly Merges: Disaster!
140 engineers
![Page 15: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/15.jpg)
![Page 16: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/16.jpg)
First attempt at fixing
![Page 17: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/17.jpg)
Bring in the horse power
![Page 18: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/18.jpg)
Internal Open Source Model
![Page 19: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/19.jpg)
Splitting the Repository
![Page 20: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/20.jpg)
Hmmmm… NuGets!
![Page 21: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/21.jpg)
Problems
![Page 22: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/22.jpg)
Repositories everywhere
2012 => 33 repositories2013 => 160+ repositories
![Page 23: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/23.jpg)
Spaghetti and NuGetscriteo.frontend
criteo.display
criteo.cookies
common.utils
criteo.display
criteo.urlscriteo.objectscriteo.rendering
criteo.bidding
criteo.serve
criteo.sql
criteo.data
criteo.banners
criteo.memcache
flash
criteo.biere
![Page 24: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/24.jpg)
Repository OwnershipAll repos to be owned by a teamMerge requests for repos you don’t own
![Page 25: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/25.jpg)
Component Teams
![Page 26: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/26.jpg)
Conway’s Vengeance“Organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations.”
Teams dealt with unclear interfaces, animosity grew stronger.
![Page 27: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/27.jpg)
Propagating changes
![Page 28: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/28.jpg)
SOX compliance (an example)Required updating the SDK (lowest level repository)Program Manager full time for over one monthSome TLAs were 5 SDK versions behind
![Page 29: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/29.jpg)
Release problem
Code Freeze during peak season:Development continued
When code unfroze:One month worth of code to integrateTook 5 months to release again
![Page 30: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/30.jpg)
It wasn’t scaling!Team had almost doubled
TLAs could take over ayear to release next time
We could lose our abilityto release forever
![Page 31: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/31.jpg)
So why did we fail ?
developers owners of application
ae64ca3
57d21a9
4567a81
9aad4cb
f478ff347ac58171da3b5
![Page 32: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/32.jpg)
![Page 33: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/33.jpg)
Integrate early
developers
57d21a94567a81 9aad4cbf478ff3 47ac58171da3b5
![Page 34: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/34.jpg)
Trunk based development
![Page 35: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/35.jpg)
Trunk based development
![Page 36: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/36.jpg)
TBD is a real pain...
“Branch by Abstraction" is a technique for making a large-
scale change to a software system in gradual way that
allows you to release the system regularly while the
change is still in-progress.
Martin Fowler
![Page 37: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/37.jpg)
But we love it
![Page 38: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/38.jpg)
What did it take ?
![Page 39: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/39.jpg)
Build infrastructure
Virtual team for 6 months.
Afterwards, team became
permanent.
![Page 40: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/40.jpg)
Killing the NuGet lag
● Move every git repository to the new code review tool
● Revert to the last version in production
● Build master continuously and show progress
● Deploy in preproduction then production
![Page 41: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/41.jpg)
Test plans progress
![Page 42: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/42.jpg)
Our setupin great (technical) details
![Page 43: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/43.jpg)
Code reviews with gerritMOAB
![Page 44: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/44.jpg)
...
...
...
Building and publishing project 70783/70784 : XXX ... [OK]
Building and publishing project 70784/70784 : XXX.UTest... [OK]
70784 projects successfully built (100.0 %).
10:09:12 cbs assembly-set update --moabId 13998
assemblyset.json written.
10:09:20 cbs export
Build 6a306a43dd147cd6a6fcacbf40e20b25f3f69845 exported.
10:10:23 cbs assembly-set upload
Assemblyset pushed to [email protected]:qa/moab.git
10:10:31 cbs tag HEAD build/current
The MOAB
![Page 45: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/45.jpg)
The MOAB pipeline
![Page 46: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/46.jpg)
The sandboxesa datacenter
a sandbox
![Page 47: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/47.jpg)
How we deploydatacenter
1. scp
2. bittorrent
![Page 48: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/48.jpg)
How we deploy
![Page 49: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/49.jpg)
Automatic metrics checks
![Page 50: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/50.jpg)
cost of change
number of devs *
Conclusion
* including 5% working full time on engineering tools
![Page 51: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/51.jpg)
We’re not the only ones
![Page 52: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/52.jpg)
Mandatory “we’re hiring” slide
![Page 53: Delivery at Scale](https://reader036.fdocuments.us/reader036/viewer/2022062313/559b21aa1a28abc9738b47b1/html5/thumbnails/53.jpg)
Questions ?