November 15 cloud bees clusterhq meetup fli, flockerhub, and jenkins
-
Upload
ryan-wallner -
Category
Software
-
view
206 -
download
0
Transcript of November 15 cloud bees clusterhq meetup fli, flockerhub, and jenkins
![Page 1: November 15 cloud bees clusterhq meetup fli, flockerhub, and jenkins](https://reader031.fdocuments.us/reader031/viewer/2022030307/58e5d8891a28ab1d608b58f7/html5/thumbnails/1.jpg)
Let’s talk CI/CD and DevOpswith Cloudbees & ClusterHQ
Ryan WallnerDeveloper Evangelist, ClusterHQ
![Page 2: November 15 cloud bees clusterhq meetup fli, flockerhub, and jenkins](https://reader031.fdocuments.us/reader031/viewer/2022030307/58e5d8891a28ab1d608b58f7/html5/thumbnails/2.jpg)
What’s in here?
How to get CI and Staging
to use realistic data
for each branch and commit from every developer
![Page 3: November 15 cloud bees clusterhq meetup fli, flockerhub, and jenkins](https://reader031.fdocuments.us/reader031/viewer/2022030307/58e5d8891a28ab1d608b58f7/html5/thumbnails/3.jpg)
![Page 4: November 15 cloud bees clusterhq meetup fli, flockerhub, and jenkins](https://reader031.fdocuments.us/reader031/viewer/2022030307/58e5d8891a28ab1d608b58f7/html5/thumbnails/4.jpg)
Docker and Jenkins today● Jenkins is great for automating builds … but it’s not so great at
managing data, this is left to external systems
● Docker and Docker Compose are great for defining and running multi or single container environments … not so true for managing data.
● How do you pull point-in-time references of data to anywhere in your CI/CD pipeline?○ Consider Dev, Test, Staging, QA and Production.
![Page 5: November 15 cloud bees clusterhq meetup fli, flockerhub, and jenkins](https://reader031.fdocuments.us/reader031/viewer/2022030307/58e5d8891a28ab1d608b58f7/html5/thumbnails/5.jpg)
Example Jenkins Workflows
![Page 6: November 15 cloud bees clusterhq meetup fli, flockerhub, and jenkins](https://reader031.fdocuments.us/reader031/viewer/2022030307/58e5d8891a28ab1d608b58f7/html5/thumbnails/6.jpg)
Jenkins Master
Jenkins Slave + Docker
Jenkins Slave + Docker
Jenkins Slave + Docker
Docker Staging
$ git push origin branch
Notify
Build Build
Success?
Dev Test Staging
*or maybe only when you merge into master or specific staging branches
![Page 7: November 15 cloud bees clusterhq meetup fli, flockerhub, and jenkins](https://reader031.fdocuments.us/reader031/viewer/2022030307/58e5d8891a28ab1d608b58f7/html5/thumbnails/7.jpg)
But.. staging and testing without realistic data isn’t great● Does it represent what a user would see?
● Is there “enough” data to see how the application works under load?
● Is the data isolated so I can test or see how staging looks with different types or amounts of data?
● Do I know where this data came from, who created it?
![Page 8: November 15 cloud bees clusterhq meetup fli, flockerhub, and jenkins](https://reader031.fdocuments.us/reader031/viewer/2022030307/58e5d8891a28ab1d608b58f7/html5/thumbnails/8.jpg)
![Page 9: November 15 cloud bees clusterhq meetup fli, flockerhub, and jenkins](https://reader031.fdocuments.us/reader031/viewer/2022030307/58e5d8891a28ab1d608b58f7/html5/thumbnails/9.jpg)
FlockerHub and FliFlockerHub is GitHub for data
a. Hub & Catalog for Docker data volumesb. Fine-grained access controls & user mgmtc. Available initially as ClusterHQ hosted
service
Fli is Git for data
a. cli interface for FlockerHubb. Snapshot Docker volumesc. Push & pull from FlockerHubd. Available as Apache 2.0 download
![Page 10: November 15 cloud bees clusterhq meetup fli, flockerhub, and jenkins](https://reader031.fdocuments.us/reader031/viewer/2022030307/58e5d8891a28ab1d608b58f7/html5/thumbnails/10.jpg)
User View
Data Servers
https://flockerhub.clusterhq.com
https://data.flockerhub.clusterhq.com
Github/Google
Data operations
Session
AuthTokentoken.txt
![Page 11: November 15 cloud bees clusterhq meetup fli, flockerhub, and jenkins](https://reader031.fdocuments.us/reader031/viewer/2022030307/58e5d8891a28ab1d608b58f7/html5/thumbnails/11.jpg)
Staging / test with realistic data is great - w/ FlockerHub● Does it represent what a user would see?
Yes, I can have prod ops give us obfuscated production snapshots so we can get a realistic view of data.
● Is there “enough” data to see how the application?Yes, we can get a fresh, up-to-date snapshot from Ops teams.
● Is the data isolated so I can see how staging looks with different types or amounts of data?
Yes, we can get fresh clones of a snapshot we choose each and every commit.
● Do I know where this data came from, who created it?Yes, We can track who the snapshot was created by and who has access to it.
![Page 12: November 15 cloud bees clusterhq meetup fli, flockerhub, and jenkins](https://reader031.fdocuments.us/reader031/viewer/2022030307/58e5d8891a28ab1d608b58f7/html5/thumbnails/12.jpg)
Example Jenkins Workflow with Fli and FlockerHub
![Page 13: November 15 cloud bees clusterhq meetup fli, flockerhub, and jenkins](https://reader031.fdocuments.us/reader031/viewer/2022030307/58e5d8891a28ab1d608b58f7/html5/thumbnails/13.jpg)
Jenkins Master
Jenkins Slave + Docker
Jenkins Slave + Docker
Jenkins Slave + Docker
Docker Staging
$ git push my-branch
Notify
Build Build
Success?
Dev Test Staging
*Jenkinsfile has a reference of what Snapshot to use in staging.
$ fli pull
*every developer, every branch. Isolated snapshots
pushpull
![Page 14: November 15 cloud bees clusterhq meetup fli, flockerhub, and jenkins](https://reader031.fdocuments.us/reader031/viewer/2022030307/58e5d8891a28ab1d608b58f7/html5/thumbnails/14.jpg)
Jenkins Master
Jenkins Slave + Docker
Docker Staging
$ git push my-branch
Notify
Build test_123With snapshot 321
Success?
Dev Test Staging
*Jenkinsfile has a reference to use snapshot 321 for test 123
$ fli pull
1. pull (only changes)2. clone snapshot3. give clone to docker4. docker run && test5. snapshot volume if failure6. docker stop/kill
![Page 15: November 15 cloud bees clusterhq meetup fli, flockerhub, and jenkins](https://reader031.fdocuments.us/reader031/viewer/2022030307/58e5d8891a28ab1d608b58f7/html5/thumbnails/15.jpg)
Manage snapshots, orgs and teams https://flockerhub.clusterhq.com
![Page 16: November 15 cloud bees clusterhq meetup fli, flockerhub, and jenkins](https://reader031.fdocuments.us/reader031/viewer/2022030307/58e5d8891a28ab1d608b58f7/html5/thumbnails/16.jpg)
Demo
Jenkins workflow with Fli and FlockerHub
![Page 17: November 15 cloud bees clusterhq meetup fli, flockerhub, and jenkins](https://reader031.fdocuments.us/reader031/viewer/2022030307/58e5d8891a28ab1d608b58f7/html5/thumbnails/17.jpg)
Thank YouSign up
https://clusterhq.com/register/flockerhub-beta/ @ryanwallner
![Page 18: November 15 cloud bees clusterhq meetup fli, flockerhub, and jenkins](https://reader031.fdocuments.us/reader031/viewer/2022030307/58e5d8891a28ab1d608b58f7/html5/thumbnails/18.jpg)
Walkthrough❏ Checkout new branch to work with, push and watch build in Jenkins❏ Show Staging❏ Change branch to use a different snapshot that has a bad phone number or vin number
❏ Show output from error in CI❏ Use manifest produced by failure to reproduce locally.❏ vi fli-docker-manifest.yml (in inventory-app)❏ fli-docker run -f fli-docker-manifest.yml -c -t /root/fh.token❏ docker ps to show locally running re-created env with snapshot❏ run the test that failed locally.
❏ docker run --net=flicompose_net -e FRONTEND_HOST="frontend" -e DATABASE_HOST="db" -e FRONTEND_PORT=8000 --rm -v ${PWD}/frontend/:/frontend clusterhq/mochatest "cd /frontend && npm install && mocha --debug test/test_db_vehicle_vins.js"
❏ cleanup with fli-docker destroy -f fli-docker-manifest.yml -clean
![Page 19: November 15 cloud bees clusterhq meetup fli, flockerhub, and jenkins](https://reader031.fdocuments.us/reader031/viewer/2022030307/58e5d8891a28ab1d608b58f7/html5/thumbnails/19.jpg)
Backup/Example Slides
![Page 20: November 15 cloud bees clusterhq meetup fli, flockerhub, and jenkins](https://reader031.fdocuments.us/reader031/viewer/2022030307/58e5d8891a28ab1d608b58f7/html5/thumbnails/20.jpg)
Test failures within Jenkins
![Page 21: November 15 cloud bees clusterhq meetup fli, flockerhub, and jenkins](https://reader031.fdocuments.us/reader031/viewer/2022030307/58e5d8891a28ab1d608b58f7/html5/thumbnails/21.jpg)
Test logs show failed test and what snapshot was taken after the failure
![Page 22: November 15 cloud bees clusterhq meetup fli, flockerhub, and jenkins](https://reader031.fdocuments.us/reader031/viewer/2022030307/58e5d8891a28ab1d608b58f7/html5/thumbnails/22.jpg)
Jenkins Build/Test with Snapshots/ClonesJenkins first pulls the necessary snapshots and creates a volume for the Docker app to use.[parallel tests 9] Syncing Volumeset inventory-app[parallel tests 9] Getting ID of the Snapshot 750k-records-snap[parallel tests 9] ID of snapshot is 6234bb23-15c9[parallel tests 9] Pulling inventory-app:6234bb23-15c9[parallel tests 9] Creating volume from Snapshot[parallel tests 9] Volume /chq/e41408a1-9e41/bd6f5bc0-f4ad created[parallel tests 9] Loading the path into the application.
The volume /chq/e41408a1-9e41/bd6f5bc0-f4ad/ will be mapped into the docker-compose.yml
The application boots up with a point in time snapshot of data in staging for the particular branch
* http://jenkinsdemo.clusterhq.com/job/inventory-pipeline-multi/job/master/
![Page 23: November 15 cloud bees clusterhq meetup fli, flockerhub, and jenkins](https://reader031.fdocuments.us/reader031/viewer/2022030307/58e5d8891a28ab1d608b58f7/html5/thumbnails/23.jpg)
Staging is available per commit with its own isolated data.
![Page 24: November 15 cloud bees clusterhq meetup fli, flockerhub, and jenkins](https://reader031.fdocuments.us/reader031/viewer/2022030307/58e5d8891a28ab1d608b58f7/html5/thumbnails/24.jpg)
fli-dockerhttps://github.com/ClusterHQ/fli-docker
![Page 25: November 15 cloud bees clusterhq meetup fli, flockerhub, and jenkins](https://reader031.fdocuments.us/reader031/viewer/2022030307/58e5d8891a28ab1d608b58f7/html5/thumbnails/25.jpg)
Stateful-Apps as Code?fli-docker
The fli-docker utility is designed to simplify the deployment of stateful applications inside Docker containers using a Flocker Hub Stateful Application Manifest (SAM) used side by side with the Docker Compose file.
![Page 26: November 15 cloud bees clusterhq meetup fli, flockerhub, and jenkins](https://reader031.fdocuments.us/reader031/viewer/2022030307/58e5d8891a28ab1d608b58f7/html5/thumbnails/26.jpg)
In a local development environment$ cat docker-compose.yml
services: db: networks: - net image: rethinkdb volumes: - rethink-data:/data ports: - 28015 - 29015 - 8080 command: rethinkdb
$ cat fli-manifest.yml
docker_app: docker-compose.yml
flocker_hub: endpoint: data.flockerhub.clusterhq.com
volumes: - name: rethink-data snapshot: someSnapshot-of-DB volumeset: inventory-app