Comparing Agile QA Approaches to End-to-End Testing

Comparing Agile QA Approaches to End-to-End Testing

Daria Mehra, Director of QE at Quid

Talk at Lean Startup Week, 11/03/16

Why test at all?Cant we write bug-free software?Industry Average: "about 15 - 50 errors per 1000 lines of delivered code.(Steve McConnell, Code Complete)Cant we fix the bugs as the users report them?Are you Facebook?Cost of fixing the bug goes up with time from inceptionPrevent it at no cost*, fix it pre-release for $, fix it post-release for $$$$.Hope is not a strategy. (Google SRE)

Buggy mathMercedes recently ran an ad saying that its new car has 100 million lines of code. Thats impressiveuntil you realize that on average, any piece of software has a few bugs per thousand lines of code, so the car probably has thousands of vulnerabilities. Chris Dixon of Andreessen Horowitz, replying to Obamas 6 tech challenges100 mln loc * 15 bugs/kloc => 1.5 mln bugs!Ford F-150 pickup, 150 mln loc => 2.25 mln bugsAssume 0.001% probability of bug in a given monthGet 1 bug per working day

How to move fastRemember Move fast and break things?Slogan v2: Move fast and build things!Because you dont move faster with a broken leg.Quality vs Velocity is a false dichotomyQuality improves as waste is removed, leading to velocityKaizen: change for the betterWhat Agile was called in 1986

Bimodal IT: where do you land?

http://techbeacon.com/gartners-bimodal-it-mistake-devops-can-deliver-velocity-qualityY u so slow?Dont do thatFast wont lastAwesome. How?

Path to quality

Velocity: High.Except for when it breaks. Quality: ?????Velocity: High.Quality: High.https://danashby.co.uk/2016/10/19/continuous-testing-in-devops/

Who needs end-to-end tests?

http://www.alwaysagileconsulting.com/articles/end-to-end-testing-considered-harmful/https://testing.googleblog.com/2015/04/just-say-no-to-more-end-to-end-tests.htmlEnd-to-end testing considered harmful. If thats all you do. (Steve Smith)Unit tests are great! At Quid: 5000+ unit tests, 108 e2e user-workflow tests.

2 unit tests, 0 integration tests

If this video wont play in the slide deck, watch it here: http://thisotplife.tumblr.com/post/137415092034/2-unit-tests-0-integration-tests

needs end-to-end tests!Whats Quid? Quid is a platform that searches, analyzes and visualizes the worlds collective intelligence to help answer strategic questionsHuge data set + upload your ownData-dependent executionSearch -> NLP -> knowledge network -> vizRich UI with many workflows:Lots of ways to slice and dice the dataImport and export and browsers, oh my!Must test important user workflows daily in CI

Two roads divergedManual-only testing is not a feasible path: slow, expensive, unreliable.Two ways Ive done it: full automation with Selenium, and RainforestQA.

Selenium approachAutomated end-to-end testcases in Python / Ruby / etc, executed by driving browser actions via Selenium frameworkOften with a layer of structured natural language on top (Gherkin or Cucumber)Pros:Fast execution -> quick feedback loopRuns in CI on developer branches and master -> always releasable masterCons:Tied to the front-end implementation, your CSS is now your APIFinicky, tends to suffer from timing issues with page stateHard to debug failures, is this a product regression or a test issue? Element is not clickable at point (x, y). Other element would receive the click:Misses regressions that human would noticeTakes engineering skill to add and maintain testcases

Release process w/Selenium

Unit Tests + Functional Tests + End-to-End Selenium Tests

RainforestQA approachAutomated end-to-end testcases in unstructured natural language (English),executed by crowdsourced human testers via RainforestQA servicePros:Easy to add and maintain testcases, non-engineers can do itEasy to debug test failures (screenshot + human comment)No dependence on front-end implementation, only user-visible workflowHuman eyeParallel scalable executionCons:Slower test execution at human speedAnnual contract expenseRuns on staging and/or production, not on dev branches

Release process w/RainforestQA

Unit Tests + Functional TestsEnd-to-End RainforestQA Tests

Why we picked RainforestQANot either/or:Quid uses Selenium in production monitoring checksScaling out Rainforest testcases was SO MUCH EASIERSolved regression testing for a complex productNightly testing is close enough to CI/CD for usNext up: early access testing with feature flagsAllows QE team to stay smallPromotes collaboration on quality between PM, CS, QE and DevFrees up Quality Engineer time for artisanal exploratory testing And that has made all the difference.

Ask me anythingDaria Mehra, Director of QE at QuidLook us up: www.quid.comEmail me: dmehra@quid.comRead my post on quality process: http://tinyurl.com/dmehra-ikeaRead about how I spend my days: http://tinyurl.com/dmehra-dayAnd we ARE hiring!