Testing in a continuous delivery environment
-
Upload
stefan-verhoeff -
Category
Technology
-
view
413 -
download
0
description
Transcript of Testing in a continuous delivery environment
Testing in a Continuous Delivery
EnvironmentSiddharth Somasundaram
Stefan Verhoeff
HERE
● Formerly known as Nokia / NAVTEQ Maps
● Powers Bing, Facebook and Yahoo Maps
● 4 out of every 5 cars with in-dash navigation are
powered by HERE
Testing Before Agile
● Testers are only involved in the later stage
● Long regression testing cycle before release
● Developers and Testers have separate work streams
○ Leads to conflicts between them
● Business pressure to release the software
○ Compromise on quality?
Importance of Agile
● Requirements change over time
● Testers are part of the planning process
● The entire team is responsible for Quality
● Faster release cycle
○ From once every 3 months to daily releases
Continuous Delivery
● Release early and often to production
● Automate testing and deployment
● Faster feedback, reduce risks and overhead for releasing
● Hot trend in the industry, especially web companies
But what about Quality Assurance?
● Lots of automation for regression testing
● Build enough trust in the test coverage
● Focus shifts from prevention of bugs to rapid discovery
and resolution
● Exploratory testing can be decoupled from releasing
How do we do it?● Team Culture and Technical practices we found to be useful
Team Culture Practices
● One agile Team, One Product and One Goal
○ Product Owner
○ Scrum Master
○ Developers / Testers
○ Offshore teams
● Communication solves most of the problems
Collaborate using Work Agreements
The team formulates rules and hold each other
accountable. For example:
● Running tests before committing
● No commits on red build
● First priority is to keep the build green
Build Monitor visible in every room
Pair Programming
Share knowledge with Tech Talks
Combine developer and operation teams
Technical Practices
Continuous Integration
Test Driven Development● Code and tests are developed at the same time
Build Pipeline
●
Test Coverage at all levels
© Alister Scott http://watirmelon.com/2011/06/10/yet-another-software-testing-pyramid/
Eliminate Flaky Tests
● Essential to keep the team
trusting the test results
● Mocking dependencies
Feature Flags
● No branching
○ Prevent integration problems
○ Quality built in
● Turn features on in production
when they are ready
● Quality boundaries for
different release stages
Logging & Monitoring
● What happens after we go live?
● Production monitoring ensures edge cases are covered
● Real-time metrics for: system, application, business
Logging & Monitoring
● Quickly responding to alerts
● Production is the ultimate proof the code works
Bug introduced We fixed it :-)
In Conclusion...
Is it Successful? The Metrics
2010 Now
Release quarterly Release daily
Mostly manual testing 7000+ unit tests, 400+ GUI tests
Preparing release takes weeks
Release package ready in 45 minutes
Deployment takes days Deploy 3 sites in 10 minutes
Management signoff required
Team empowered to make changes
Deployments 2012 vs 2013
Business Benefits
● Reduce risk involved in big changes
● Reduce time to market
● Smaller more frequent feature rollouts
● Quickly learn from feedback on product changes
Business Benefits
● Predictable releases
● Improved team morale
○ Less time spend in stressful firefighting
○ Less bug fixing, more feature delivery
○ Satisfaction of delivering to the customer
Challenges
● Everybody agreeing on the process
● Discipline comes over time only
● Feedback needs to be given quickly
● High investment in test automation and keeping it
running smoothly
● Need to react to incidents quickly
Future Plans
● Deploy every build to production
● Keep on investing in Automation and Monitoring
● Canary deployments to validate new releases before
global rollout
● Leverage quick cycle time for A/B testing driven feature
development
HERE ChecklistTeam Culture
● One Team
● Work Agreements
● Build Monitor in the
room
● Pair Programming
● Tech Talks
● DevOps
Technical
● Continuous Integration
● Test Driven Development
● Build Pipeline
● Test Coverage on all
levels
● Feature Flags
● Logging & Monitoring
Merci BeaucoupThank You
Danke SchönGrazie