Bring Your Project From a 10 Years to a 3 Months Release Cycle
-
Upload
samsung-open-source-group -
Category
Technology
-
view
79 -
download
0
description
Transcript of Bring Your Project From a 10 Years to a 3 Months Release Cycle
Bring Your Project From a 10 Years to a 3 Months Release Cycle
Stefan Schmidt
Agenda
Introduction
Release History
Technological Side
Social Influences
Next Steps
Summary
Introduction
Enlightenment is not a web project
Instead it is written in plain C
Enlightenment is not developed by a StartUp
Instead it is developed by the FOSS community and companies such as Samsung
Enlightenment is not developed by a 5 person team in one office
Instead it is developed by an international team distributed over many timezones
All this might result in different understandings of agile development :-)
Release History
Release History
The Enlightenment project started in 1997 by Carsten Haitzler
Some early releases until E 0.16
A 12 years gap from first code to release E 0.17
Building a whole new set of libraries for it but no releases for them either for a long time
More release driven now but still feature based and not time based schedule
Reasoning
Most of the active developers at that time did not see a need for releases
It was a project driven by volunteers which used the code straight out of CVS/SVN
Some argued for it as the fastest agile development cycle ;-)
Nobody stood up for taking care of the releases so it did not get done
What Changed?
Different people joined the project which had more interested in releases
Most Linux distributions wanted us to do releases before including our code
Commercial parties also asked for releases
With the pressure also the understanding grow that releases would help the project forward
How did we do it?
That is what the rest of the talk is about
From the technological side we improved our tools and infrastructure
On the social side we steadily worked on giving releases a more prominent role
Technological Side
Tools and Automation
One of the biggest roadblocks is to build trust in the code base you are going to release
In the Enlightenment project with the two main libraries alone having around 850.000 lines of code
You need a variation of tools that will help you to build up this trust
SCM, automated builds and maybe continuous integration, static analyzers just to name a few
SCM and Understandable Commits
As an very old project CVS was used in the beginning
Switch to SVN for a newer tool but same old practices
Large amounts of work committed as one change
Many unrelated changes at once
Switch to git
After some struggling it helped to allow people working on different things without dumping everything into master
Improved overall commits, code as well as commit messages
People start to develop larger changes in branches
Automated Builds
Setup automated builds with Jenkins
Quick builds for every change pushed pushed to master
Building for x86, x86_64, x32 with gcc as well as clang and also cross-compilation for windows with mingw
Very useful as many developer only use one system for their work
Many of the discovered problems are getting fixed instantly (see social pressure)
In the dark times before we had this setup builds have been broken for some configurations for months
During the nightly builds we run time intensive things like static
analyzers and other testing
Static Analyzer
Static analyzer can be very helpful for a plain C project
Started with scan-build, part of clang, from the LLVM project
Output quite noise and sadly a lot false positives
Also experimented with klocwork but the proprietary nature made it hard to share results with the community
Now also using Coverity which is also proprietary but offers a free service for FOSS projects
Especially Coverity uncovered quite some issues which have been overlooked by code review
You still have to deal with false positives but the signal to noise ratio is way better
Test Suite
We have unit testing for a lot of components
But with 358 tests we only cover 29.4% of our lines of code
We are an UI toolkit so work is also ongoing on pixel perfect regression testing
We are still weak on the automated testing side and need to improve
One of the bigger worries before every release
Build Complexity
By history the Enlightenment project was happy to add configuration options
Over the last two years we tried to reduce these options to something maintainable
Merged libs allow easier maintenance. 11 different libs have been merged into EFL
All of these actions had direct influence on how much different configurations we can test
The reduced set of libraries also eased the release process
Social Influences
General Social Changes
Once we had IRC and email notification about broken builds social pressure grow to fix them
Keep people in the loop and highlight positive changes (e.g. weekly QA newsletter)
Most people have been ok with adapting to changes but you have to provide docs and guidance
People follow the lead but would not have done it on their own. (e.g. backports of fixes to stable branches)
Towards Releases
Easier once we came over a long time without a release
Getting into the habit to do releases more often
Core application now do releases a while after the libraries
Happened naturally
Switch to time based releases. Now fine tuning the process to identify problems
Took over as release manager (Only because I build up trust to the code base due to all the automation)
There need to be proof that it will really work
Invest time to make a sane and realistic schedule
Next Steps
Changes to come
ABI checker to avoid breakage surprises after a release
Jenkins job that builds a magic developer branch
Allowing developers to tests their changes for many configurations
Thinking about adding arm builds to jenkins
Thinking about running automated valgrind runs to detect memory issues (-fsanitize* could be an alternative)
Fine tuning of the release process
Think about regression test. Having them mandatory for fixes might blow our man power.
Integrate the pixel perfect regression tester with jenkins for fast feedback
Changes that might never come
Every commit runs through our jenkins testing before it goes into master (Qt project does this)
More code review for core developers before changes hit the repo
Hard to bring in as many developers want to avoid any delay before the code hits the repo
Automate the release process (NEWS generation, announcement templates, upload tarballs, etc)
A simple version bump and pushed git tag might be enough to get it all done :-)
Summary – What you should take home
Automate as much as possible to keep the workload down and the project always in a good state
Invest in all kind of tools to make your job easier
Show the benefits of more regular releases and proof that it is possible
Keep on fine tuning the process to avoid as much problems as possible
Expect that people will only follow if you give them the lead and show how the process could work
Thank you.
References
http://www.enlightenment.org
http://build.enlightenment.org
http://git.enlightenment.org
Imagery sources
http://www.linuxtag.org/2014/
http://www.enlightenment.org/
https://openclipart.org/detail/192629/gear-tools-by-ben-192629
http://git-scm.com/downloads/logos
http://jenkins-ci.org/
http://www.coverity.com/
https://openclipart.org/detail/26498/present-blue-pack-by-minduka