2014QA Process Overview

5.14.2014

These Wonderful People!Who is your QA?

Roles QA Plays• Product Coverage

• Platform Coverage

• Stability and Performance

• Community

Roles QA Plays (cont..)• Assist development with Automation and Tree maintenance

• Active bug triage

• Device Flashing and Maintenance

• Partner and User support

• Internal Testing programs (Beta)

Product Coverage• QA representative per functional team

• Building and running Test cases on features based on user stories

• Daily coverage

• Bug analysis across branches (triage, qawanted, regression hunting)

• Automated and manual smoketests on device

• Hitting acceptance criteria

Device Automation

• Setting up 30 flame devices for lab automation

• Executing various tests: Gaia functional, power measurement, b2g performance (fps and cold launch), and marketplace

• Reporting to Jenkins server short term, long term Task cluster by way of Tree Herder

User story planning Acceptance defined

U definedTestcase creation Regression tests

cleanup

Partial Feature Testrun Regression TR1

Bug Bash Feature review FL Acceptance

Regression TR2 L10n TR1

MTBF Features exploratory Regression TR3

Bug Bash FC acceptance

Daily smoke tests / Ongoing Bugwork

!L10n TR2

Partner CS/ IOT stuff Internal certifications

CC acceptance

Testrun Metrics• 1.3 Case study

! •! % test coverage completed! ! ◦! run1 = 87% completed  (12/09/2013)! ! ▪! 3360 passed, 413 failed, 218 invalid, 154 skipped  = 4145! !! ◦! run2 = 98% completed! ! ▪! 3721 passed, 323 failed, 96 invalid, 0 skipped = 4104! !! ◦! run3 - 99% completed! ! ▪! 4034 passed, 262 failed, 121 invalid, - skipped = 4415! !! •! # of blockers found in test run  (triaged for 1.3+)! ! ◦! run1 = 27 ! ! ◦! run2 = 20 ! ! ◦! run3 = 7! !! •! Test Failures / (#of tests * %completed) = Test Failure %! ! ◦! run1 = 413 / (4145 * 0.87) = 11.45%! ! ◦! run2 = 323 / (4140 * 0.98) = 7.96%! ! ◦! run3 = 262 / (4415 * 0.99) = 6.00%!

Testrun Metrics (cont)• 1.3 Exploratory

! •! # of blockers in exploratory run  [search whiteboard = dogfood1.3]! ! ◦! 41 bugs  (http://goo.gl/3dNUfl)! ! •! # of man hours on exploratory! ! ◦! 2 Weeks of 1/27 - ~11 testers! ! ◦! 40 hour week! ! •! Blockers per man hours = (# of testers * # hours / total # of blockers)! ! ◦! (11* 80 hours) / 41 blockers = 2.15 blockers! !!

Acceptance Criteria

• Product coverage Criteria listed for FL, FC, and CC milestones

• Platform and automation milestones are ongoing

• Tracked in: https://wiki.mozilla.org/Release_Management/FirefoxOS/Release_Milestones

• Working with different platform teams for backlog of test coverage (ie. WebRTC, graphics)

• Building out existing Mochitest coverage for Gfx / JS / Dom / Layout / Web Api

• Building test apps on device for Product QA to execute

Platform Coverage

• Q2 Goal: 72 hours uptime per device, no crash/ no hangs

• Running Hamachi (1.4) and Flame (2.0) [10 hamachis in parallel]

• Latest results (April 29th, 2014)

• Buri - 12 hours uptime per device

• Flame - 50 hours uptime per device

• Executed once a week, starting after FL

Stability (MTBF)

• Perf Metrics. establish a in-house baseline. analyzing results and building more automated tests

• Working draft: https://wiki.mozilla.org/FirefoxOS/Performance/Release_Acceptance

• Results through Datazilla and Eideticker tools

• Security testing? Seccomp builds? B2G fuzzing? working on these things in the future

Perf and Security

• With Flame, working with Foxtrot and other teams on focus areas

• Posting entry projects for device and automation tasks on public pages (oneanddone, badges)

• Bug bashes, local meetups

• Mentoring / code reviewing process

• More transparency with tests, reports, bugs, and testing opportunities

Community

Find us!• in your meetings, workweeks, offices, bugs, right behind you.

• https://wiki.mozilla.org/B2G/QA

• irc #fxosqa