Testing Tools and Jenkins - European Southern Observatory · PDF fileTesting Tools and Jenkins...
Transcript of Testing Tools and Jenkins - European Southern Observatory · PDF fileTesting Tools and Jenkins...
![Page 1: Testing Tools and Jenkins - European Southern Observatory · PDF fileTesting Tools and Jenkins ... Running static analysis with Cppcheck. ... genhtml -o coverage_html](https://reader031.fdocuments.us/reader031/viewer/2022022502/5aacb2117f8b9a8d678d34d9/html5/thumbnails/1.jpg)
ESO Pipeline Workshop, Oct 24 2014
on behalf of
ESO Pipeline Systems Group
Testing Tools and Jenkins
Artur Szostak
![Page 2: Testing Tools and Jenkins - European Southern Observatory · PDF fileTesting Tools and Jenkins ... Running static analysis with Cppcheck. ... genhtml -o coverage_html](https://reader031.fdocuments.us/reader031/viewer/2022022502/5aacb2117f8b9a8d678d34d9/html5/thumbnails/2.jpg)
ESO Pipeline Workshop, Oct 24 2014
Purpose
● Using a number of testing tools to identify problems and bugs in the software.
● Integrated into the Jenkins continuous integration system to run these tools over the code base in a regular and systematic manner.
● Give feedback to the developers as fast as possible when problems/bugs are detected.
● Use the tools to guide development of unit and regression tests.● Help assess the quality of the code.● Provide facilities for developers to extend or add their own
testing jobs quickly and easily.
![Page 3: Testing Tools and Jenkins - European Southern Observatory · PDF fileTesting Tools and Jenkins ... Running static analysis with Cppcheck. ... genhtml -o coverage_html](https://reader031.fdocuments.us/reader031/viewer/2022022502/5aacb2117f8b9a8d678d34d9/html5/thumbnails/3.jpg)
ESO Pipeline Workshop, Oct 24 2014
Jenkins – Continuous Integration
● Standard jobs are maintained in Jenkins for all pipelines, CPL, HDRL and esorex.
● These are run regularly and automatically when changes are detected in the source code repository.
● Users can also create customised build and test jobs which can be triggered either manually or automatically.
![Page 4: Testing Tools and Jenkins - European Southern Observatory · PDF fileTesting Tools and Jenkins ... Running static analysis with Cppcheck. ... genhtml -o coverage_html](https://reader031.fdocuments.us/reader031/viewer/2022022502/5aacb2117f8b9a8d678d34d9/html5/thumbnails/4.jpg)
ESO Pipeline Workshop, Oct 24 2014
Tests Covered in Jenkins
● Testing compilation and linking on a number of platforms.● Check for compiler and linker warnings.
● Running unit tests.● Running static analysis with Cppcheck.
● Coverity Scan is also used for CPL.● Various additional sanity checks for naming conventions etc. are
performed with the staticcheck script.● Running regression tests using esorex on test data sets.● Running unit tests under Valgrind to check for memory leaks and
errors.● Code coverage for unit tests and regression tests.● Creation and testing installation of pipeline kits.
![Page 5: Testing Tools and Jenkins - European Southern Observatory · PDF fileTesting Tools and Jenkins ... Running static analysis with Cppcheck. ... genhtml -o coverage_html](https://reader031.fdocuments.us/reader031/viewer/2022022502/5aacb2117f8b9a8d678d34d9/html5/thumbnails/5.jpg)
ESO Pipeline Workshop, Oct 24 2014
Jenkins Results
● Summary information available per Pipeline.● Lots of additional information available in GUI.
![Page 6: Testing Tools and Jenkins - European Southern Observatory · PDF fileTesting Tools and Jenkins ... Running static analysis with Cppcheck. ... genhtml -o coverage_html](https://reader031.fdocuments.us/reader031/viewer/2022022502/5aacb2117f8b9a8d678d34d9/html5/thumbnails/6.jpg)
ESO Pipeline Workshop, Oct 24 2014
Warnings and Unit Tests Example
![Page 7: Testing Tools and Jenkins - European Southern Observatory · PDF fileTesting Tools and Jenkins ... Running static analysis with Cppcheck. ... genhtml -o coverage_html](https://reader031.fdocuments.us/reader031/viewer/2022022502/5aacb2117f8b9a8d678d34d9/html5/thumbnails/7.jpg)
ESO Pipeline Workshop, Oct 24 2014
Cppcheck Results Example
![Page 8: Testing Tools and Jenkins - European Southern Observatory · PDF fileTesting Tools and Jenkins ... Running static analysis with Cppcheck. ... genhtml -o coverage_html](https://reader031.fdocuments.us/reader031/viewer/2022022502/5aacb2117f8b9a8d678d34d9/html5/thumbnails/8.jpg)
ESO Pipeline Workshop, Oct 24 2014
Staticcheck Script Example
![Page 9: Testing Tools and Jenkins - European Southern Observatory · PDF fileTesting Tools and Jenkins ... Running static analysis with Cppcheck. ... genhtml -o coverage_html](https://reader031.fdocuments.us/reader031/viewer/2022022502/5aacb2117f8b9a8d678d34d9/html5/thumbnails/9.jpg)
ESO Pipeline Workshop, Oct 24 2014
Valgrind Results Example
![Page 10: Testing Tools and Jenkins - European Southern Observatory · PDF fileTesting Tools and Jenkins ... Running static analysis with Cppcheck. ... genhtml -o coverage_html](https://reader031.fdocuments.us/reader031/viewer/2022022502/5aacb2117f8b9a8d678d34d9/html5/thumbnails/10.jpg)
ESO Pipeline Workshop, Oct 24 2014
Coverage Results Example
![Page 11: Testing Tools and Jenkins - European Southern Observatory · PDF fileTesting Tools and Jenkins ... Running static analysis with Cppcheck. ... genhtml -o coverage_html](https://reader031.fdocuments.us/reader031/viewer/2022022502/5aacb2117f8b9a8d678d34d9/html5/thumbnails/11.jpg)
ESO Pipeline Workshop, Oct 24 2014
Coverage Results Example
![Page 12: Testing Tools and Jenkins - European Southern Observatory · PDF fileTesting Tools and Jenkins ... Running static analysis with Cppcheck. ... genhtml -o coverage_html](https://reader031.fdocuments.us/reader031/viewer/2022022502/5aacb2117f8b9a8d678d34d9/html5/thumbnails/12.jpg)
ESO Pipeline Workshop, Oct 24 2014
Conclusion
● Using Jenkins in Pipeline Group for ~ 1.5 years with positive feedback from developers.
● Integrating testing and static analysis tools into Jenkins made the tools:
● easier to use,● and used more often.
● Automated testing with Jenkins (or equivalent system) is the only feasible way to perform systematic large scale testing.
● Consortia are encouraged to run Jenkins like setups for regular testing of the code during development.
![Page 13: Testing Tools and Jenkins - European Southern Observatory · PDF fileTesting Tools and Jenkins ... Running static analysis with Cppcheck. ... genhtml -o coverage_html](https://reader031.fdocuments.us/reader031/viewer/2022022502/5aacb2117f8b9a8d678d34d9/html5/thumbnails/13.jpg)
ESO Pipeline Workshop, Oct 24 2014
Backup slides...
![Page 14: Testing Tools and Jenkins - European Southern Observatory · PDF fileTesting Tools and Jenkins ... Running static analysis with Cppcheck. ... genhtml -o coverage_html](https://reader031.fdocuments.us/reader031/viewer/2022022502/5aacb2117f8b9a8d678d34d9/html5/thumbnails/14.jpg)
ESO Pipeline Workshop, Oct 24 2014
Sample Cppcheck Commands
URL=http://svnhq2.hq.eso.org/p2/trunk/Pipelines/iiinstrument/iiinstrumentp
svn co $URL
cd iiinstrumentp
find . -type f -name "*.h" | xargs -L 1 dirname \
| sort -u > paths.txt
cppcheck --force --enable=all --std=c99 --std=posix \
--includes-file=paths.txt ./
![Page 15: Testing Tools and Jenkins - European Southern Observatory · PDF fileTesting Tools and Jenkins ... Running static analysis with Cppcheck. ... genhtml -o coverage_html](https://reader031.fdocuments.us/reader031/viewer/2022022502/5aacb2117f8b9a8d678d34d9/html5/thumbnails/15.jpg)
ESO Pipeline Workshop, Oct 24 2014
Commands for staticcheck
./bootstrap
./configure --enable-maintainer-mode \
--prefix=`pwd`/testOutput
make clean
make install
make check TESTS=
URL=http://svnhq2.hq.eso.org/p2/trunk/Pipelines/ci-jenkins/scripts/staticcheck
svn export $URL
./staticcheck -namespace=iiinstrument -strict \
-extlibs=cpl,hdrl -excludes='\./hdrl/\.libs/.*,\./irplib/\.libs/.*'
![Page 16: Testing Tools and Jenkins - European Southern Observatory · PDF fileTesting Tools and Jenkins ... Running static analysis with Cppcheck. ... genhtml -o coverage_html](https://reader031.fdocuments.us/reader031/viewer/2022022502/5aacb2117f8b9a8d678d34d9/html5/thumbnails/16.jpg)
ESO Pipeline Workshop, Oct 24 2014
Sample Valgrind Commands
./configure --enable-maintainer-mode \
--prefix=`pwd`/testOutput
make clean
make install
make check TESTS=
cd iiinstrument/tests
valgrind --leak-check=full --track-origins=yes \
.libs/iiinstrument_dfs-test
![Page 17: Testing Tools and Jenkins - European Southern Observatory · PDF fileTesting Tools and Jenkins ... Running static analysis with Cppcheck. ... genhtml -o coverage_html](https://reader031.fdocuments.us/reader031/viewer/2022022502/5aacb2117f8b9a8d678d34d9/html5/thumbnails/17.jpg)
ESO Pipeline Workshop, Oct 24 2014
Commands for Code Coverage
./configure --prefix=`pwd`/testOutput \
CFLAGS='-g -O0 --coverage' \
LDFLAGS='-g -O0 --coverage'
make clean
make install
find . -name "*.gcda" -delete
make check
lcov --compat-libtool --directory ./ -c -o coverage.info
genhtml -o coverage_html coverage.info
firefox coverage_html/index.html
![Page 18: Testing Tools and Jenkins - European Southern Observatory · PDF fileTesting Tools and Jenkins ... Running static analysis with Cppcheck. ... genhtml -o coverage_html](https://reader031.fdocuments.us/reader031/viewer/2022022502/5aacb2117f8b9a8d678d34d9/html5/thumbnails/18.jpg)
ESO Pipeline Workshop, Oct 24 2014
Jenkins – Setup
● Master running on 24 core machine with 64GB RAM and RAID storage.
● Only ~ 1% CPU used by Jenkins master when idle. Up to 10-15% during peak load.
● About 16GB used by Jenkins JavaVM instance.● 2.5TB partition dedicated for Jenkins build logs. Currently using ~
45% of this.● 16 physical and 36 virtual build slaves running various
versions of Apple OS X, Scientific Linux, Fedora, Debian, Ubuntu, OpenSUSE, CentOS.
● Running ~ 750 standard jobs and ~ 80 customised user jobs for 25 pipelines, CPL, HDRL, esorex and dependencies.