Red Hat on Jenkins: By the numbers

Jenkins User Conference San Francisco, Oct 2nd 2011 #jenkinsconf

Red Hat on Jenkins: By the numbers

Nam DuongRed Hat, Inc.

https://openshift.redhat.com


Agenda

Products currently leveraging JenkinsDeployment size: # of Masters/Slaves/Jobs in use across Jboss Middleware QETop 10 list of interesting PluginsCode contributionsExperiences/Pitfalls/Lessons learnedBest PracticesDemo:

Production/Staging server (sheer numbers)OpenShift on Jenkins

EC2 and GIT plugin demo


Red Hat products

Products currently leveraging JenkinsJBoss offerings:

Enterprise Application PlatformJBoss Developer StudioSOAJBoss PortalJBoss OSGIMany, many others

OpenShift – OpenShift is our Platform as a Service (PaaS) offering. Our Jenkins instances are running in the cloud, hosted on OpenShift Flex and used to build and test OpenShift


Deployment Size

JBoss Middleware Team:– 1 Master

• IBM System x3550 M3, 2x E5640 CPU, 24GB RAM, 2x SAS 73GB 15K-RPM in RAID1

– Slaves connected 113 (various OS: linux, Win, Solaris, HP, AIX)

– ~4700 Jobs configured (~1000 running/month)– ~95% of Middleware test automation is

called/executed via JenkinsOpenShift Flex Team:– 1 Master

• 7.5 GB, 4CPU, 10GB EBS (RHEL 5.6) – 10 Slaves Building EC2 Images and 5 slaves

running tests – 20 jobs (RHEL 6.1)– ~90% of our automation is executed via Jenkins


Interesting plug-ins used

Management (Groovy*, Disk Usage*, Build Publisher*, Workspace Cleanup*)Cloud Control (EC2 plugin, DeltaCloud*)Enterprise focused plugins (SmartFrog)Code Coverage (cobertura, emma)Source control (GIT/SVN/CVS)

*Represents code contributions


Code Contributions

Memory leaks: Involved in diagnosis, resolutionIn the Hudson timeframe, we contributed to stability through reporting of bugs/instability (large deployment)First ones to run into these types of problems (pure volume)LTS (long term support - Jenkins stable branch)Development and maintenance of plugins (see previous)reporting bug + bug fixes


Experiences/Pitfalls/Lessons learned

1 master may not handle all jobs at once - potentially provide failover functionality on JenkinsJobs getting stuck due to polling console output through ssh. Need to limit jobs from writing to console.Disk Utilization issues - Overcome through quotas set on jobsLogging was too verbose, Jenkins instance stalled due to low/no disk spaceOOM issues when trying to view the giant logs - fix in progress


Best Practices

Set limits on memory usage (quota) of slaveSet limits (quota) on Disk usage for jobsTools on shared NFS drive – for maintenance and replication/recoveryMemory – lots of memory to buffer incremental growth of MasterJenkins master and all slaves under CFEngine control (config management/recovery)Like all reliable services, use Dev/Stg/Prod increments. Can scale vertically, but has anyone tried scaling horizontally?


Demo

Production instance within corporate firewall:Production: <Screenshots available upon request>

Staging our pluginsStaging: <Screenshots available upon request>

Jenkins running on OpenShift Flex; testing OpenShift Flex

<Screenshots available upon request> EC2 and GIT plugin demo

Used to create a client testbedCreates an EC2 instance, prints the IP address, pulls down all code bits


Links/Contact

Email: [email protected]://openshift.redhat.com

We appreciate your feedback of your user experience

https://openshift.redhat.com/


Thank You To Our SponsorsPlatinumSponsor

GoldSponsor

SilverSponsor

BronzeSponsors

Coming Soon: The CloudBees Newsletter for Jenkins

Please complete the Jenkins survey to help us better serve the community

(bonus: a chance to win an Apple TV!)

Red Hat on Jenkins: By the numbers

Documents

Transcript of Red Hat on Jenkins: By the numbers