7 Years In TestingTamás Tompa (Digital Natives)

tamas.tompa@digitalnatives.hu

Budapest Ruby Meetup 2013

11/09/2013

Thursday, September 12, 13

§ 7 years in ruby, rails§ 7 years in webapp testing§ 5 years in agile development§ 5 years in automated testing

Thursday, September 12, 13

Looking for testing best practices

Thursday, September 12, 13

Short version

INTEGRATEAUTOMATE

Thursday, September 12, 13

Short version

INTEGRATEAUTOMATE(everything)

Thursday, September 12, 13

Human VS Machine§ Use humans for thinking§ Use machines for executing

§ Humans can forgive, forget things§ Machines are strict, consequent

Thursday, September 12, 13

Theory

Thursday, September 12, 13

Development Workflow

Resolved Deployed

Testing / Working

Testing / Not

working

In progress Stable

for feature based development

Thursday, September 12, 13

Automating steps

Resolved Deployed

Testing / Working

Testing / Not

working

In progress Stable

Manual Automatic Manual Automatic

Thursday, September 12, 13

Automated Testing§ rspec§ turnip§ cucumber§ capybara§ selenium -> phantomjs§ factory girl§ jenkins / cloud CI

TOOLS

Thursday, September 12, 13

Working, bug free software is priceless...

Thursday, September 12, 13

Working, bug free software is more expensive...

Thursday, September 12, 13

... but only if you don’t know how to test

and maintain your code

Thursday, September 12, 13

Write tests for those parts of the code which are...

§ the most often used§ changing frequently§ changing data and working with

financial data§ more likely to fail

Thursday, September 12, 13

Instrumentation§ Instrumentation means that the code gets enhanced with

flags, and when the execution passes a certain flag, it gets set. When a test code coverage measurement is made then the tool:§ instruments the code - places these flags§ runs the test cases - flags get set§ and finally prints out the result - how many flags have

been set

§ During usage coverage measurement, test execution is replaced by customer interaction.

Thursday, September 12, 13

Most often used code§ Instrument the code base and deliver

it to the testers / customer§ Turn on coverage measurement in your

(staging) application§ Deploy it§ Do manual testing of critical functions§ Summarize and analyze metrics§ Write tests for highly covered classes

Thursday, September 12, 13

Frequently changing code§ Use version control system logs

§ Gitinspector§ StatSVN

§ Find out which files change often!

Thursday, September 12, 13

Working with data§ Make code reviews§ Re-prioritize classes and methods

§ Add more priority when data is critical§ Move to top when working with money

Thursday, September 12, 13

Failing, problematic code§ Find and analyze bugs in your Issue

Tracker System§ Write test before fixing the issue§ Review CI server logs

§ collect often failing tests§ Refactor unstable code

§ refactor your tests also

Thursday, September 12, 13

Git branching§ Work only with feature branches§ Test the branches separately§ Use Pull Requests to integrate

feature branches to a development branch

§ Do more (manual) testing on development branch

§ Merge easily to stable, than finally to production

Thursday, September 12, 13

Methods§ TDD - it’s up to you :)§ BDD - based on TDD + domain

specific acceptance criterias§ CDD - for prototypes, spikes

Thursday, September 12, 13

In practice...

Thursday, September 12, 13

Manual Testing§ Plan your test cases

§ at Sprint Planning Meetings§ Allocate a good Tester to the team

§ If you can’t, test all User Story after completion, within team, but between team members

§ Try to measure coverage during manual testing

Thursday, September 12, 13

Continuous Integration§ Jenkins server§ Cloud alternatives: travis-ci, circleci, drone.io§ Parallel job execution (queues / executors)§ Main branches VS Feature branches§ Instant builds VS Daily builds§ Conflicting databases and shared resources§ CI rake task in the project§ Re-run logic for Cucumber tests§ Long running jobs

Thursday, September 12, 13

CI - Ruby, Rails§ Jenkins / RVM Plugin

§ RVM-managed environment:ruby-1.9.3-p392@PROJECTNAME

§ Jenkins / Ruby Metrics Plugin§ Rcov coverage report§ Other stats

§ Bundler integration

Thursday, September 12, 13

Thursday, September 12, 13

CI - Feature branches§ Jenkins / Git Plugin

§ Branches to build: ^(?!.*master|stable|development|origin\/HEAD).*$

§ Jenkins / Version Number Plugin§ Version Number Format String:

#${BUILD_NUMBER}-${GIT_BRANCH}

Thursday, September 12, 13

CI - Build types§ Instant builds

§ Higher priority§ Immediately after commit§ Build trigger: Poll SCM

§ Daily builds§ Once a day§ Long running tests§ Build trigger: Build periodically

Thursday, September 12, 13

CI - Rake

task :commit => :clean do Rake::Task['db:migrate'].invoke Rake::Task['db:test:load'].invoke Rake::Task['db:test:purge'].invoke Rake::Task['ci:build:rspec_internal'].invoke Rake::Task['ci:build:cucumber_internal'].invokeend

rake ci:build:commit

> Example: https://gist.github.com/tompata/6513021

Thursday, September 12, 13

CI - Rerun§ Re-run only previously failing tests§ Re-try the randomly failing tests§ Built-in support in Cucumber§ Easy integration with Jenkins§ Save resources and time

> Example: https://gist.github.com/tompata/6513060

Thursday, September 12, 13

CI - Integration§ Bitbucket§ Redmine§ Notifiers (HipChat, Email, Desktop)§ Deployment (Capistrano)§ Monitoring (Airbrake, New Relic)

Thursday, September 12, 13

Thursday, September 12, 13

Thursday, September 12, 13

Thursday, September 12, 13

Thursday, September 12, 13

Thursday, September 12, 13

Browser tests§ integration tests

§ cucumber / capybara§ responsive design

§ phantomjs / screen sizes§ cross browser (design)

§ mtest (npm) - multiplatform§ saucelabs / cloud

§ 162 device/os/browser platforms

Thursday, September 12, 13

Responsive design§ PhantomJS§ Screen resolutions (window_size)§ Cucumber profiles

> Example: https://gist.github.com/tompata/6512905

Thursday, September 12, 13

Stress tests§ penetration / load testing before

releases§ use separated, dedicated

environments (staging test)§ use virtual instances (ec2)§ monitor instances (new relic)§ use online tools (loadimpact,

loader.io) or local (jmeter)

Thursday, September 12, 13

Coverage, Quality§ rcov§ simple cov§ the magical 80%

§ code climate - code analysis, quality

Thursday, September 12, 13

INTEGRATEAUTOMATE

Thursday, September 12, 13

Resources§ http://zsoltfabok.com/blog/2011/05/narrow-down-what-to-test/

§ http://nvie.com/posts/a-successful-git-branching-model/

§ http://jenkins-ci.org/

§ https://wiki.jenkins-ci.org/display/JENKINS/Ruby+Metrics+Plugin

§ https://saucelabs.com/§ https://github.com/digitalnatives/

§ https://github.com/digitalnatives/mtest

§ https://gist.github.com/tompata/6513021

§ https://gist.github.com/tompata/6513060

§ https://gist.github.com/tompata/6512905

§ https://codeclimate.com/

§ http://loadimpact.com/

Thursday, September 12, 13

We’re hiring!http://digitalnatives.hu/jobs

Thursday, September 12, 13