Continuous Integration: Improving Quality in Moodle
-
Upload
dan-poltawski -
Category
Technology
-
view
413 -
download
2
Transcript of Continuous Integration: Improving Quality in Moodle
Continuous Integration: Improving Quality in Moodle
Dan Poltawski Integrator
Moodle HQ
@dan_p
the world’s open source learning platform
#mootieuk15
the world’s open source learning platform
Wide topic, today focusing on:
• “Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible.” - Martin Fowler
http://www.martinfowler.com/articles/continuousIntegration.html
Continuous Integration
the world’s open source learning platform
Integration Process: 2001 - 2010Numerous problems:
• Single developer may be only person aware of a
change
• Inconsistency was rife
• Developers often unaware of the consequences of
some changes
• Testing was not enforced
• Unit tests were limited and frequently broken
New Development
Release
the world’s open source learning platform
Moved to git and brought about a change in
development process
• Enforced code review
• Enforced ‘manual testing’
• Integration team introduced to move
towards ensuring more consistency
• Introduction of CI systems
Integration Process: Today
Automated Tests
Automated Checks
New Development
Peer Review
Integration Review
Testing
Release
the world’s open source learning platform
• Needed automated tests to supplement the manual reviews done by the Integration team
• CI Server was setup - Jenkins • Open source/extensible • Self hosted • http://jenkins-ci.org
Post-integration checker
Automated Tests
Automated Checks
New Development
Peer Review
Integration Review
Testing
Release
the world’s open source learning platform
Post-integration checkerJenkins polls for changes in
integration.gitCode pushed to integration.git
Issue status is set (e.g. failed)
Detects code changes and starts running checked jobs
Jenkins notifies status changes
Integration team Jenkins
the world’s open source learning platform
Post-Integration Checks
the world’s open source learning platform
Post-integration checks: whitespace• First job to be created (the whitespace police
started it all!)
• Detect incorrectly spacing added
• Frequent error: • Easy to miss setting up your editor for
Moodle • Not spotted easily by reviewers
the world’s open source learning platform
Post-integration checks: compare upgrade & install
• Detects DB inconsistencies between install and upgrade code:
1. Runs the installer 2. Run upgrade from a previous
version 3. Compares the database
schema from both 4. Fails in the case of any
differences
• Finds schema inconsistencies previously frequently undetected - (common source of number of 1.9 -> 2.x upgrade bugs)
• Also brings fringe benefit of testing upgrade code (though limited by lack of data)
the world’s open source learning platform
• Initially support for Moodle 2.0-2.2 with SimpleTest
• Custom CLI generated test data, ran tests and output results as JUnit xml file
• Culturally testing was not embedded • Tests rarely run or added by developers • Had to mock everything - no way to
setup test data
Post-integration checks: Unit testing
the world’s open source learning platform
• Moodle 2.3 added support for PHPUnit
• Data generators significantly improved unit testing experience
• Helped move cultural change towards automated tests
• Challenges: • Keeping the test time down • ‘Random’ fails
Post-integration checks: Unit testing
Moodle 2.3 Moodle 2.6 Moodle 2.9
24,273 44,491 58,011
the world’s open source learning platform
Unit test assertions
the world’s open source learning platform
• Many created a result of mistakes slipping through: • Incorrect upgrade savepoints
• version.php file check
• Check files built correctly (less, shifter, grunt)
• php syntax check (php -l)
• Unresolved merge conflicts
Post-integration checks: Others
the world’s open source learning platform
• Desirable to check before integration • Can help reduce delays
(timezones) by providing more rapid feedback
• Better suited to more pedantic checks
• Provide supplemental information rather than just pass/fail
Pre-integration checker
Automated Tests
Automated Checks
New Development
Peer Review
Integration Review
Testing
Release
Inline comments must end
in full-stops, exclamation marks, or question
marks.
the world’s open source learning platform
Pre-integration checker
Merges branch with moodle.git
Run pre check jobs against merged branch
Produce ‘smurffile’ of changes, filtered by lines affected by patch
jshint
phpcs
csslint
phpdoc
php -l
commit
savepoint
thirdpartyReport summary back to tracker
checkstyle.xml
the world’s open source learning platform
• Not possible to run all our testing scenarios on a per-commit basis due to time to run
• Nightly server does these long-running tasks • Behat (Acceptance tests) • Multi-platform tests
• phpunit: pgsql, mariadb, mysql, mssql, oracle
• behat: firefox, chrome, phantomjs
Nightly checks
Automated Tests
Automated Checks
New Development
Peer Review
Integration Review
Testing
Release
the world’s open source learning platform
//Teacher creates a discussion @Test public void teacherCreateDiscussion() throws Exception { //Adding a discussion forum.clickForumLink(this.properties.get("nameOfForum")); forum.clickAddNewDiscussionTopicButton(); forumPosts.enterSubjectField(this.properties.get("teacherSubject")); forumPosts.enterMessage(this.properties.get("teacherMessage")); forumPosts.clickPostToForum(); }
Automated UI tests: Selenium Native
• Java-based selenium test suite • Not embedded in development process
• Tests written by test manager, not developers • Maintained in separate repository
Scenarios in 2.3
28
the world’s open source learning platform
Given I add a "Forum" to section "1" and I fill the form with: | Forum name | Test forum name | | Forum type | Standard forum for general use | | Description | Test forum description | | Subscription mode | Optional subscription | And I add a new discussion to "Test forum name" forum with: | Subject | Test post subject | | Message | Test post message |
Automated UI tests: Behat
• Improving from pure selenium implementation • No java :-) Human-friendly ‘Gherkin’ language • PHP integration code • Tests required to land with features by developers
• Culturally embedded Scenarios in 2.9
412
the world’s open source learning platform
• Many pre-checker ideas
• Run phpunit/behat/multi-
platform tests in pre-checker
(split by component)
• Intelligent re-checking when
branches get updated
• Backup/restore checks
• Custom jira fields for prechecker
• Automated rejections
• Webservices testing
• Run all post-check’s in pre-checker
• Others as we detect them
Future
Questions?
@dan_p
the world’s open source learning platform
https://github.com/moodlehq/moodle-local_ci