Continuous Integration on Steroids
-
Upload
alexander-akbashev -
Category
Engineering
-
view
204 -
download
3
Transcript of Continuous Integration on Steroids
![Page 1: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/1.jpg)
Continuous Integration on SteroidsAkbashev AlexanderHighload++ | November 07, 2016
![Page 2: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/2.jpg)
![Page 3: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/3.jpg)
Agenda
01. CI in HERE 02. Monitoring 03. Scalability 04. Jenkins 05. Nightmares Plugins 06. Morale 07. Q&A
![Page 4: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/4.jpg)
01Continuous Integration in HERE
![Page 5: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/5.jpg)
Every change goes through validation pipeline
Gerrit Gerrit Plugin
Pre-submit Trigger
Pre-submit Trigger
Build
Build
Build
Build
Build
TestsTestsTestsTestsTestsTests
TestsTestsTestsTests
TestsTests
Tests
![Page 6: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/6.jpg)
Feedback goes from tests back to Gerrit
Gerrit Gerrit Plugin
Pre-submit Trigger
Pre-submit Trigger
Build
Build
Build
Build
Build
TestsTestsTestsTestsTestsTests
TestsTestsTestsTests
TestsTests
Tests
![Page 7: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/7.jpg)
Feedback comes from every pipeline
Gerrit Gerrit Plugin
Pre-submit Trigger
Pre-submit Trigger
Build
Build
Build
Build
Build
TestsTestsTestsTestsTestsTests
TestsTestsTestsTests
TestsTests
Tests
![Page 8: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/8.jpg)
Numbers
100k+ builds per day ~1.5k concurrent builds 1.3-2.5k executors
• Each “build” is execution of one build/test job
• Total number correlates with number of commits
• Number of builds is not so important as number of commits
• Big throughput is extremely important
• Morning commit • Before lunch • “Last attempt for today”
• Raised on-demand • Health checks • Jenkins strategy is not
optimized for cloud
![Page 9: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/9.jpg)
02Monitoring
![Page 10: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/10.jpg)
Collects information about every build in system
Groovy Event
Listener Plugin
Jenkins build Fluentd InfluxDB Grafana
![Page 11: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/11.jpg)
Collects information about every build in system
Groovy Event
Listener Plugin
Jenkins build Fluentd InfluxDB Grafana
![Page 12: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/12.jpg)
JVM stats are the best “canary”
Groovy Event
Listener Plugin
Jenkins build Fluentd InfluxDB Grafana
Jenkins JVM
![Page 13: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/13.jpg)
![Page 14: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/14.jpg)
03Scalability
![Page 15: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/15.jpg)
What do we want to achieve?
![Page 16: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/16.jpg)
What do we want to achieve?
Keep feedback time (< 20 min.)
![Page 17: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/17.jpg)
What do we want to achieve?
Keep feedback time (< 20 min.)Test as much as possible
![Page 18: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/18.jpg)
What do we want to achieve?
Keep feedback time (< 20 min.)Test as much as possible… with debug symbols
![Page 19: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/19.jpg)
What do we want to achieve?
Keep feedback time (< 20 min.)Test as much as possible… with debug symbols… and code coverage information
![Page 20: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/20.jpg)
What do we want to achieve?
Keep feedback time (< 20 min.)Test as much as possible… with debug symbols… and code coverage informationand on physical devices
![Page 21: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/21.jpg)
How to scale
Increase number of executors Minimize job execution time Smart testing
![Page 22: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/22.jpg)
How to increase number of executors?
EC2 Plugin TestDroid
![Page 23: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/23.jpg)
How to minimize job execution time
![Page 24: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/24.jpg)
How to minimize job execution time
Split tests by type
![Page 25: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/25.jpg)
How to minimize job execution time
Split tests by typeParallel execution
![Page 26: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/26.jpg)
How to minimize job execution time
Split tests by typeParallel executionNode as cache storage
![Page 27: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/27.jpg)
How to minimize job execution time
Split tests by typeParallel executionNode as cache storageShared compiler cache
![Page 28: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/28.jpg)
How to minimize job execution time
Split tests by typeParallel executionNode as cache storageShared compiler cacheProfiling!
![Page 29: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/29.jpg)
04Jenkins
![Page 30: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/30.jpg)
Is Jenkins so slow or we are doing something wrong?
![Page 31: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/31.jpg)
Is Jenkins so slow or we are doing something wrong?
Jenkins is ok.
![Page 32: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/32.jpg)
Is Jenkins so slow or we are doing something wrong?
Jenkins is ok.But…
![Page 33: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/33.jpg)
Surprise #1
Rotation costs a lot
![Page 34: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/34.jpg)
Surprise #2
It works much better with nginx
less jenkins.access.log | tail -n1000 | grep urt=\"\-\" | wc -l407
![Page 35: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/35.jpg)
Surprise #3
Some buttons are very dangerous
![Page 36: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/36.jpg)
Surprise #3
Some buttons are very dangerous
![Page 37: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/37.jpg)
Slave
Slave
One fundamental issue
Master
Slave
Slave
Slave
Slave
Slave
Slave
Users
![Page 38: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/38.jpg)
What can you find in heap dump of OOM-Killed Jenkins?
![Page 39: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/39.jpg)
What can you find in heap dump of OOM-Killed Jenkins?
Console logs
![Page 40: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/40.jpg)
Console logs
Should be less than X MB Verbose output goes to file “>” and “tee” are amazing!
![Page 41: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/41.jpg)
What can you find in heap dump of OOM-Killed Jenkins?
Console logs
![Page 42: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/42.jpg)
What can you find in heap dump of OOM-Killed Jenkins?
Console logs Build history
![Page 43: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/43.jpg)
Build history
2000 entities or 3 days Efficient rotator
![Page 44: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/44.jpg)
What can you find in heap dump of OOM-Killed Jenkins?
Console logs Build history
![Page 45: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/45.jpg)
What can you find in heap dump of OOM-Killed Jenkins?
Console logs Build history Build artifacts
![Page 46: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/46.jpg)
Build artifacts
Push to S3 directly from slaves Don’t store anything on master
![Page 47: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/47.jpg)
05Nightmares Jenkins Plugins
![Page 48: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/48.jpg)
Limit of number of builds
20K
![Page 49: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/49.jpg)
Groovy Event Listener Plugin
all events synchronized groovy compilation
fixed since 1.010 (Mar 10, 2016)
![Page 50: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/50.jpg)
Limit of number of builds
40K
![Page 51: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/51.jpg)
Warnings Plugin
Just another parser of console log
parseConsole is “deprecated” parseFile is allowed 0 warnings are very appreciated :)
![Page 52: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/52.jpg)
Limit of number of builds
60K
![Page 53: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/53.jpg)
Timestamper Plugin
Tail needs not only “tail”
fixed since 1.8.5 (Aug 31, 2016)
![Page 54: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/54.jpg)
Limit of number of builds
60K
![Page 55: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/55.jpg)
EC2 Plugin
Full list of all images in AWS
fixed since 1.35 (Jun 30, 2016)
![Page 56: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/56.jpg)
Limit of number of builds
90K
![Page 57: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/57.jpg)
Robot Framework Plugin
Green chart costs 100 times more
Replaced by xUnit Plugin
![Page 58: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/58.jpg)
Limit of number of builds
120K
![Page 59: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/59.jpg)
Build Failure Analyzer Plugin
One regexp One stream One thread
PR-57 is not accepted yet
![Page 60: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/60.jpg)
Limit of number of builds
140K
![Page 61: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/61.jpg)
Cleanup Workspace Plugin
`ü` breaks everything
PR-29 is not accepted yet
![Page 62: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/62.jpg)
06Morale
![Page 63: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/63.jpg)
Final recommendations
![Page 64: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/64.jpg)
Final recommendations
Think about scalability at first place
![Page 65: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/65.jpg)
Final recommendations
Think about scalability at first placeFlakiness could be a huge problem
![Page 66: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/66.jpg)
Final recommendations
Think about scalability at first placeFlakiness could be a huge problemReduce memory allocations
![Page 67: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/67.jpg)
Final recommendations
Think about scalability at first placeFlakiness could be a huge problemReduce memory allocationsCache as much as possible
![Page 68: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/68.jpg)
Final recommendations
Think about scalability at first placeFlakiness could be a huge problemReduce memory allocationsCache as much as possibleFailing builds can be expensive
![Page 69: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/69.jpg)
Workflow
Slowness? Profile! Fix! Contribute!
![Page 70: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/70.jpg)
Open source collaboration
Let’s make our life better ;)
![Page 71: Continuous Integration on Steroids](https://reader034.fdocuments.us/reader034/viewer/2022051300/5874f7631a28ab917a8b82bf/html5/thumbnails/71.jpg)
Full list of our contributions related to this talk
• Jenkins • ccache • clcache • EC2 Plugin • S3 Plugin • FluentD Plugin
• BuildRotator Plugin • Groovy Event Listener Plugin • Timestamper Plugin • Robot Framework Plugin • Build Failure Analyzer Plugin • JVM GC Log Plugin for
FluentD