Mozilla Web QA: Who, What, Why, How

Stephen Donner + Team

Brownbag

March 12, 2015

Web QA: Who

• Stephen Donner• Krupa Raj• Rebecca Billings• Matt Brandt• Bob Silverberg• Victor Carciu• Madalin Cotetiu• Valentina Peleskei• (Dave Hunt)• And dozens of amazing community members!

Web QA: What

• Actively supported projects:• Marketplace• AMO• Mozilla.org• Brand campaigns• SUMO• Socorro• BIDPOM (BrowserID Page-Object Model)• MozTrap• One and Done

• Projects for which we have test-automation:• Affiliates• Mozillians• Snippets• Wiki.mozilla.org• Bouncer (download.mozilla.org)• Webmaker

Web QA: Why

• Because websites/apps don’t test themselves (much)• and developers don’t or can’t always test enough, either• Team scale:

• ~ 11 to 1 ratio (dev -> QA)• Because coverage/value-add:

• True full-stack:• Config is code• Real browsers• “Real” user accounts• Real networks

• load-balancers, CDNs, DB replication, caching...• ...real problems (!)

• Because we have Open Badges and contributor paths:• https://badges.mozilla.org/en-US/profiles/profile/webqa.badges• https://wiki.mozilla.org/QA/Execution/Web_Testing/Contributor_levels

• OH: “Web QA knew and reported the site was down even before Nagios tripped and alerted us” - anonymous Web Ops

Web QA: How

Manual testing

● Feature verification, cross-browser and cross-environment testing, usability testing, l10n testing, exploratory testing, specification testing, stress and load testing, etc

Automation

● The end-to-end test repositoryo Running, observing test results, and maintaining current testso Identification and creation of new coverage.

● Reviewing pull request● Historically Web QA has used git issues to track automation needs● Review how to interact with the channel-bot -- webqatestbot● Review how to access/use https://webqa-ci.mozilla.com/● Test repository; transition into the project's code base or continue to exist where it

currently is

Web QA: How

Other

● Managing the test plan and reviewing the risk heuristicso Reviewing components of the test plan - manual and test automation strategyo Baking community into the test plan

● Working with, reaching out to community, and maintaining relationshipso Organizing test days, events, etc

● Monitoring #moc and NewRelic + receiving tracebacks from Sentry● Triaging Bugzilla - unconfirmed/resolved/assigned/reopened/etc bugs + the

technical debt list● Documentation - maintaining up-to-date documentation [manual testing, test

automation, getting involved], blog posts, etc● Ownership of oneanddone.mozilla.org tasks

Web QA: How

• Continuous Deployment• testing game-changer (for us, industry)

• Stack:• Python (mostly); dabbling in JavaScript test-frameworks

• requests• pytest

• pytest-xdist• pytest-mozwebqa

• Selenium WebDriver• Page-Object Model (POM)• Appium

• Platforms:• Firefox:

• desktop• OS

• Android stock browser• Internet Explorer• Chrome

• Test Environments:• dev• stage• prod

Web QA: How (cont’d)

• Infra:• Continuous Integration:

• Public Jenkins: https://webqa-ci.mozilla.com/• Travis CI (Flake8)

• Both in-house Selenium Grid lab + cloud-run Selenium• Redundancy• Capacity• Flexibility

• Monitoring:• Errormill (Sentry)• New Relic

• (A few) Tools:• Fuzzing/security testing:

• OWASP ZAP• PowerFuzzer• Netsparker

Web QA: On the Horizon

• Ever-closer integration with development• process/workflows• test activity/test-results visibility• shared repositories• tooling/frameworks

• moar JavaScript (augmenting/replacing our Python?)• Integrated security testing/reporting

• https://github.com/davehunt/pytest-zap

Web QA: Where

• QMO: https://quality.mozilla.org/teams/web-qa/• One and Done exploratory-testing:

https://oneanddone.mozilla.org/en-US/tasks/available/?search=&team=6

• Web QA Wiki: https://wiki.mozilla.org/QA/Execution/Web_Testing

• Public mailing list: mozwebqa@mozilla.org• Blog: https://blog.mozilla.org/webqa/• IRC: #mozwebqa on irc.mozilla.org