Openshift for Continuous Integration Alex Lossent – IT/PES/IS AI meeting 1-Oct-2015 Openshift for...
-
Upload
thomasina-houston -
Category
Documents
-
view
224 -
download
0
Transcript of Openshift for Continuous Integration Alex Lossent – IT/PES/IS AI meeting 1-Oct-2015 Openshift for...
Openshift for Continuous Integration 1
Openshift for Continuous Integration
Alex Lossent – IT/PES/IS
AI meeting 1-Oct-2015
Openshift for Continuous Integration 2
Outline• Continuous Integration: context and history• Platform-as-a-Service concept• Docker/Kubernetes/Openshift• How Openshift compares to Puppet, Heat• Plans and outlook
AI meeting 1-Oct-2015
Openshift for Continuous Integration 3
Why a Continuous Integration platform?
• Concept: improve code quality by integrating code frequently and applying automatic quality control (tests etc.)
• Awareness of multiple CI setups for all large software projects at CERN• In IT, PH, GS, EN…• Using Bamboo, Jenkins, TeamCity…
• First experience with Jenkins in PES: AIQA• Project to provide a Jenkins “platform as a
service” • to make it easier for developers to deploy CI
AI meeting 1-Oct-2015
Openshift for Continuous Integration 4
CI: current offer• Since end 2014, we provide a CI “platform”
• Using well-established, open-source Jenkins• A Java web application
• Request an instance from http://cern.ch/forge • Documentation http://cern.ch/jenkinsdoc
• Clear interest from developers• >40 instances as of September 2015• Developers@CERN forum on 28-29 September
• A “platform” rather than a “service”• We provide the infrastructure (Puppet/Openstack)• Out of scope: support for Jenkins configuration itself, integration
of existing CI setups
• Full details presented at ITTF 5-Dec-2014
AI meeting 1-Oct-2015
Openshift for Continuous Integration 5
Significant challenges• Need more flexibility
• Every project has specific needs• No standard development platform at CERN
• Frequent requests for custom RPMs, libraries, OSes, plugins, integration scenarios etc.
• Need more automation• Many changes trigger a support ticket• Too much manual work for both users and ourselves
• Need better scaling• 1..n VM per project with CI• Security Team project to provide static code analysis
AI meeting 1-Oct-2015
Openshift for Continuous Integration 6
Towards a PaaS solution• We want to provide the infrastructure for CI• While giving maximum freedom to Jenkins
users• With minimum cost of entry and
maintenance effort for users• I.e. self-service, fully automated, self-repairing
• A good candidate for a PaaS solution
AI meeting 1-Oct-2015
Openshift for Continuous Integration 7
Options to deploy a web app…
AI meeting 1-Oct-2015
Less effortLess flexibilitySpecific help
More effortBestflexibilityUnspecific help
Web hosting:AFS/DFS websiteMiddleware-on-demand
SaaS:DrupalSharePoint
IaaS:OpenstackPuppetDNSLB
PaaS:Good flexibilityLimited effort
“to develop, run, and manage Web applications without the complexity of building and maintaining the infrastructure typically associated with developing and launching an app” [Wikipedia]
Openshift for Continuous Integration 8
PaaS: useful technologies• Containers
• Operating System-level virtualization:• A host system runs several isolated user spaces on top of a single kernel• Provides app isolation with less overhead than VMs• Containers can be provisioned and started in seconds
• Allow to separate maintenance of the host OS from that of the application itself
• Like VMs can stop worrying about physical hardware
• Docker• A popular container solution• An application can be packaged with all its environment into an image
• Includes all libraries, app framework, code, configuration…• Portable: can be deployed and instantiated with little constraints on a
variety of hosts
AI meeting 1-Oct-2015
Openshift for Continuous Integration 9
PaaS: useful technologies• Kubernetes
• Docker container orchestration system by Google• Software-defined network (SDN) to provide
advanced services• Recent but gets a lot of traction
• RedHat Openshift• Builds a PaaS solution on top of Kubernetes• Adds user management, UI, web proxy, image
registry, application templates etc.• And an application lifecycle (Continuous
Deployment)
AI meeting 1-Oct-2015
Openshift for Continuous Integration 10
What can we expect from using Openshift?
• Flexibility• Delegate full control on the Jenkins app to users
• Self-service and automation (by design)• Scaling
• Better resource-efficiency with containers
• Advanced level of service (from Kubernetes)• Persistent storage via Ceph (RBD)• Fast recovery (virtual IPs, replication controller)• Good availability (container migration)
AI meeting 1-Oct-2015
Openshift for Continuous Integration 11
How to apply this to CI?• We provide and maintain a reference Jenkins
Docker image• With relevant pre-configuration and integration• And an Openshift application template for it
• Users create instances of Jenkins on our Openshift cluster (self-service via CLI/GUI)
• Allow user-provided Docker images for slaves• Each project can have its own build environment• No need to deploy and maintain dedicated VMs• Fresh container for each job run
AI meeting 1-Oct-2015
Openshift for Continuous Integration 12
PaaS general use case• Openshift interesting for general web application hosting
• ~1400 HTTP-related openings in the external firewall• Applications currently hosted outside CERN
• More flexible than web hosting/MWoD• Wide choice of frameworks, languages…
• Significantly less effort than IaaS• No host OS to manage• Configuration management tailored for web apps
• Scalability, redundancy, good availability are almost free• Fills a gap in providing tools to facilitate web app
deployment and upgrade
AI meeting 1-Oct-2015
Openshift for Continuous Integration 13
Case study: GitLab• IaaS-based deployment
• Many VMs (dev/prod, redundancy with DNSLB)• NFS filer, DBoD (single points of failure)
• Complex configuration management with Puppet• Puppet module + tricks, custom Lemon sensors• Significant work with each new version
• Complex, manual upgrade procedure• 30 minute downtime (even for possibly online upgrades)• Not doable with Puppet (lack of synchronization), even less so
with migration of Puppet repos to GitLab
• Slow recovery in case of failure• DNS update delay• Can be much longer if DBoD or NFS filer fail (if possible)
AI meeting 1-Oct-2015
Openshift for Continuous Integration 14
Case study: what Openshift could improve
• Better resource efficiency: a lot less VMs• Direct use of Ceph (RBD) volume for storage• Single GitLab instance (“pod”) at any given time
• Faster recovery• Takes seconds to reschedule a container, reconnect to storage,
reassign virtual IP• SPoF (and dependency loop) removal: NFS filer
• possibility to host database in Openshift as well
• Considerably simplified management and procedures• Apps are described in terms of how to generate a Docker image from
source code and a deployment configuration• Openshift designed to facilitate Continuous Delivery
• Automation of building Docker images, deploying new app versions (and rollback)
• Can leverage standard RoR migration/upgrade procedure
AI meeting 1-Oct-2015
Openshift for Continuous Integration 15
Caveat: security updates• App owners responsible for maintaining their
apps… and all their dependencies• Security fixes!• Openshift provides an (optional) workflow
whereby a base image can be updated and app rebuilt on top of it
• Mostly appropriate for apps with a Continuous Deployment model• Frequent integration, updates and redeployment
AI meeting 1-Oct-2015
Openshift for Continuous Integration 16
An interesting complement to existing web app hosting offer
AI meeting 1-Oct-2015
Less effortLess flexibilitySpecific help
More effortBestflexibilityUnspecific help
Web hosting:AFS/DFS websiteMiddleware-on-demand
SaaS:DrupalSharePoint
IaaS:OpenstackPuppetDNSLB
PaaS:Good flexibilityLimited effort
Openshift for Continuous Integration 17
What about Heat?• Openstack Heat was also considered as a way to address
current challenges with CI• Heat = orchestration for Openstack• Can be used to provide a Jenkins “application template”• Improvements regarding automation, self-service, flexibility
• This option would have been fantastic if available end 2014 when CI project started
• But now suffers from availability of a solution like Openshift• Non-trivial work required:
• integration of Heat with Puppet to maintain long-lived VMs• complete refactor of how Jenkins is managed by Puppet
• Resource efficiency• “level of service” (recovery etc.)
AI meeting 1-Oct-2015
Openshift for Continuous Integration 18
Where are we now?• Prototype Openshift cluster was relatively
easy to deploy• Hosts managed with Puppet (+Ansible)• Polishing to do but no blocker in sight
• Work on a preconfigured Jenkins image in progress• With SSO, GitLab integration, etc.
• Production status before end of the year is still realistic
AI meeting 1-Oct-2015
Openshift for Continuous Integration 19
Collaboration opportunities?• Docker has been lurking around for some
time, but no well-defined strategy yet• In particular, no official SLC5/6 or CC7 images
• A new type of resource to manage• Reassign when users leave CERN, etc.• Integration with existing services?
• New technologies to master• Docker, Kubernetes are definitely technologies
that will play a major role in the coming years
AI meeting 1-Oct-2015
Openshift for Continuous Integration 20
Do you want to know more?• Openshift 3 Roadshow:
free workshop in Geneva on October 22nd
AI meeting 1-Oct-2015