Post on 10-Jan-2017
Test parallelization using Jenkins
Scott Mills - Rogue Wave Software
What am I going to cover in this session?1. About me2. About Rogue Wave3. The problem4. The challenge5. The solution6. Jenkins plugins7. What’s next8. Closing thoughts
?
What are you going to take away from this session?1. Easy to get started with Jenkins2. Monitor job health, process results3. Parallelize your test suites4. Plugins are your friends
About me
About Me
Dallas, TX
Ottawa, ON
My technical background• Over 15 years designing, developing, maintaining and supporting
Automation systems• Application Performance Testing• Application Security testing (OWASP Top 10)• Automated Web testing (Selenium, Silk Test, TestPartner)• Test Automation framework development, maintenance and
evolution in Java, Python, Jenkins
No, I don’t test software in my spare time…
About Rogue Wave Software
“Making it easier to write, test and deploy complex code”• Largest independent provider of cross-platform software
development tools and embedded components in the world• Founded in 1989• Global company, headquartered in Louisville, CO.• Customers: Over 3,000 companies in 57 countries
Tools Libraries
SourcePro OS, database, network, and analysis abstraction for C++Visualization Real-time data visualization at scale
IMSL Numerical Libraries Scalable math and statistics algorithms
PV-WAVE Visual data analysis
HydraExpress SOA/C++ modernization frameworkHostAccess Terminal emulation for WindowsStingray MFC GUI components
OpenLogic Audits Detailed open source license and security risk guidance
OpenLogic Support Enterprise-grade SLA support
Klocwork On-the-fly static code analysis for app security
TotalView for HPC Scalable parallel debugging
CodeDynamics Commercial dynamic analysis
Zend Server Enterprise PHP app serverZend Studio PHP IDEZend Guard PHP encoding and obfuscation
Rogue Wave customers
Klocwork• Continuous Static Code Analysis• Find defects early!• Web-based portal for reporting and analyzing results• Ability to be built into your Continuous Integration process• Very popular with mission critical industries• Allows customers to meet industry standards • Try it!!
The problem
Enforcing quality means ensuring quality• If Klocwork enforces code quality, Rogue Wave needs to be extremely
diligent about delivering high quality software• New features and fixes need to be tested• Regression testing extremely important – reducing false positives may
accidentally hide real code defects
?
Testing• 6000+ test cases run against our nightly build• Total execution time: 18+ hours• Too long of a delay to get results (especially in an Agile environment!)
to determine the health of the build and identify potential regression defects• Problem goes downstream: QA waits for test results, developers wait
for QA to report bugs, bugs get fixed and issue repeats.
The challenge
The challenge• Execute tests against the continuous integration build• Drastically reduce the execution time of the test suites• Reuse the existing automation framework as much as possible• Report on results as they come in• Make it scalable• Use open source solutions
The solution
The solution• Parallel execution of our tests suites across multiple VMs• Controlled by Jenkins, open source continuous integration tool• Already used to generate our nightly and CI builds• Test runs are kicked off as soon as a new CI build is available
The solution• Results imported into TestLink, open source test management system• Scalable: Add more VMs to the pool to further reduce execution time• Utilizes the bulk of the existing automation framework• High level results also sent to “Build health monitor board”
Health monitor boardJob: automation_CI
# Run Started Duration Total tests Failed Status
0 11.2.0.3913 31.08.2016 604:31 7109 77
1 11.2.0.3920 31.08.2016 441:29 7154 121
2 11.2.0.3921 01.09.2016 625:04 7043 59
3 11.2.0.3926 01.09.2016 410:19 7154 62
4 11.2.0.3928 01.09.2016 266:42 7154 60
bad
bad
good
bad
good
Health monitor board
Continuous integration• Integrating code from many developers into a baseline multiple times
a day• In best practices, paired with the execution of automated testing
upon submission
Jenkins• Open source continuous integration tool• Triggers a new build every time source code changes are submitted• Can be used for much more than just builds!• jenkins.io
Jenkins basics – demonstration• Jobs• Options• Build steps• Post-build actions
• Slaves & labels• Execution• Output• Artifacts• Downstream jobs
Solution workflowJenkins job Role Downstream jobAutomation daemon
Continuously polls an artifact produced by the CI Linux build, looking for a new build to run against (identified by a new GUID). Will run the most recent new build.
Parallelization master
Parallelization master
Inherits the data from the daemon and triggers the job matrix jobs. Aggregates the results from job matrix, reports overall build health (scripts, portal repos)
Job matrix scriptsJob matrix portal
Job matrix scripts Launches separate jobs for each individual suite of tests in the scripts repository across the Linux VMs allocated for Linux testing. Result of each suite execution are copied up to the parallelization master, and sent downstream to the TestLink import job for import into our TestLink instance.
TestLink import
Job matrix portal Launches separate jobs for each individual suite of tests in the Portal repository across the Linux VMs allocated for Linux testing. Result of each suite execution are copied up to the parallelization master, and sent downstream to the TestLink import job for import into our TestLink instance.
TestLink import
TestLink import Inherits the results.xml file from job matrix scripts and portal and imports the data into TestLink via API.
THENpython runAll.py
NOWpython parallel.py $SUITE_NAME
The queueJobs to process
scripts – config
scripts – licensing
scripts – auth
portal - reports
portal - issues
scripts - sync
portal - auth
scripts - refactor
scripts - local
portal - search
VM Farm (Label: CT_LINUX)
Ubuntu 1 : scripts - defectdetectionUbuntu 2 : freeUbuntu 3 : scripts - usabilityUbuntu 4 : portal - importUbuntu 5 : portal - buildUbuntu 6 : portal - rolesUbuntu 7 : scripts - macrosUbuntu 8 : scripts - migration
Jenkins plugins
Jenkins plugins• Many plugins available• Written by the community• Easy to install to your Jenkins instance• “I wonder if there’s a plugin for that?”
Problems solved by plugins - 1Problem:
“How can we distribute the execution of our test suites across multiple machines?”
Problems solved by plugins - 1Solution:
Matrix project plugin• The Configuration Matrix allows you to specify what steps to duplicate, and
create a multiple-axis graph of the type of build to create• User-defined axis• Slave axis
Configuration Matrix
CentOS_6.7 CentOS_7.2 Debian_7.9 Debian_8.4 Fedora_22 Fedora_23
authentication
bonobo
cdefectdetection
configuration
csharp
dbvalidator
distributed
Problems solved by plugins - 1
Pass Some failures Fail
Problems solved by plugins - 2Problem:
“Build Steps can be written in either Windows batch or Linux shell. How can we create platform independent jobs?”
Problems solved by plugins - 2Solution:
Execute Python script plugin• Our automation framework written in Python, so readily available across all
machines
Problems solved by plugins - 3Problem:
“Eeesh. If one of the slaves gets in a bad state and immediately fails tests, most tests in the queue will be directed to that slave and all fail.”
Problems solved by plugins - 3Solution:
Offline on failure plugin• Allows you to take nodes offline immediately when a job reports FAILURE
Problems solved by plugins - 4Problem:
“A job seems to be hung, it’s been running for 4 hours with no update. We’re going to have to kill it manually to free up its machine.”
Problems solved by plugins - 4Solution:
Build-timeout plugin• Abort build based on:
• Fixed time-out limit• Specific deadline time• Average duration of last n executions• No activity in log
Problems solved by plugins - 5Problem:
“Can rename the build numbers to something more applicable?”
Problems solved by plugins - 5Solution:
Build name setter plugin• Set the build name using variables, properties or file contents
Problems solved by plugins - 6Problem:
“Can we view the test results in Jenkins?”
Problems solved by plugins - 6Solution (part 1):
Junit plugin• Publish Junit test result report:
• Drill down into test suites to find individual test failures• View stack traces
Problems solved by plugins - 6
Problems solved by plugins - 6Solution (part 2):
Test results analyzer plugin• Produces a table of build history results
• Drill down into test suites to find individual test failures• Determine build-to-build trends
Problems solved by plugins - 6
What’s next?
Folks, we’re almost done…
I hope you are able to take away the following points:
1. Easy to get started with Jenkins2. Monitor job health, process results3. Parallelize your test suites4. Plugins are your friends
In closing…• Invest in your framework• Look for opportunities to evolve• Open source is great, but…• Reusing your legacy framework is great, but…
Anything else that you want to ask me?
?