Oscon2008 Qa Leak Testing Latest Slides

download Oscon2008 Qa Leak Testing Latest Slides

If you can't read please download the document

description

This is a talk that Carsten Book and I gave at OSCON about fixing memory leaks in Firefox 3.

Transcript of Oscon2008 Qa Leak Testing Latest Slides

  • 1.
      • OSCON 2008-Fixing Hard Problems Through Iterative QA and Development

Clint Talbert - Carsten Book Mozilla Corporation 24.July 2008 2. Introduction

  • Process Overview
  • Leak Testing Overview
  • Tools
  • Memory Leak Debug Test Builds
  • Leak Gauge
  • Manual Tests (Litmus)
  • Add-on Testing
  • Milestones so far

3. The Approach

  • Development and Test Development collaborate on tests and tools
  • Create widely applicable, reproducible tests
  • Test Tools are treated as first rate products
  • Implement a reporting structure for results

4. Collaboration

  • Developers and Test developers bring different perspectives to the table
  • Developers concentrate on the deep tools to measure behavior of code (i.e. Ref-Counting Analysis etc)
  • Test Developers work on harnesses and reporting infrastructure, lowering barrier of entry
  • Having reliable test tools that are run and developed on a regular basis help bring a sense of testing into the development culture.

5. Reproducible

  • Use VM's where you can
  • Keep tests and tools simple
  • By taking reliable test tools into impure systems, you can get results that you can trust.

6. Test Tools Are Products

  • Test Tools must grow with the product
  • Test Tools must be simple and maintainable
  • Test Tools must be developed and QA'd just like the product they are testing
  • Test Tools should be very easy to run
  • Test Tools should require only the most basic level of infrastructure possible to run. (But should also fit into a high infrastructure scenario for reporting)

7. Reporting Results

  • Use results to further a culture of testing
  • Use results to understand if the test is reliable, and if so broaden your testing scenarios
  • Easy to understand results empower community members

8. Reporting Results

  • What our results reporting looks like

9. Leak Testing Overview

  • Lots of Memory Improvements for Firefox 3:
  • The XPCOM cycle collector continuously cleans up unused memory. Plus, hundreds of memory leaks are now remedied
  • Memory Leak Tests after every Code change to make sure we introduce no new Memory Leaks
  • QA also started with a lot of Tests for Memory Leaks
  • Mozilla QA runs manual tests using the Litmus Testcase Suite (using Tools such as Debug Builds and Leak Gauge)
  • Regression Memory Leak Testing of Firefox
  • Memory Leak Tests of new and popular Addon's
  • Future: More Automated Tests run by Mozilla QA

10. Tools

  • Various Tools for Performance and Leak Testing
  • QA uses Debug Builds and Leak Gauge
  • Debug Builds: Debug Builds can be build with Trace-Malloc Support to search for Memory Leak Builds every time up to date build with latest checkins and overview over Leaking Components (but Debug Builds need to be built individually and you need a Development Environment (Xcode, Visual Studio Express, etc) for most platforms)
  • Leak Gauge: Developed by David Baron - It is designed to assist in detecting what leaks of large object graphs occur during normal browsing activity.The logging can be run during normal browsing without significant overhead . Log taken by setting environment variables in a release build.

11. Debug Builds

  • Using Firefox Debug Builds with Trace Malloc enabled for all Plattforms (Windows XP/Vista, Linux and Mac 10.4/10.5)
  • Trace Malloc allows searching for Memory Leaks
  • Can provide a Log File Output
  • Identify Leaking Components, this information helps Developers to debug and fix the Memory Leak !

12. Leak Gauge

  • Easy to use Memory Leak Test Tool developed by David Baron
  • Only Requirement is a environment variable and works on all platformsc:NSPR_LOG_MODULES=DOMLeak:5,DocumentLeak:5,nsDocShellLeak:5,NodeInfoManagerLeak:5
  • And set NSPR_L OG_FILE= c:leak1.log to d efine a Leak Log
  • Works on Firefox 2 and Firefox 3+ Release Builds
  • Leak Log Upload Form on http://mxr.mozilla.org/mozilla/source/tools/footprint/leak-gauge.html for analysis
  • Upload Form indicates if a Memory Leak was found

13. Leak Gauge Log Output

  • W hen a Memory Leak was found the output of the Log analysis look like this :

Memory Leak 14. Litmus based Tests

  • To make sure we have no hidden Memory Leaks in Firefox Mozilla QA was running a FFT (Full Functional Tests) Testrun in Litmus( http://litmus.mozilla.org )with Debug Builds on Linux, Mac and Windows. Memory Leaks were Reported to the Bugzilla DB.
  • Litmus is a Open Source Test Case Management Tool. Litmus contains manual testcases and a Full Functional Test contains around 750 manual testcases from all Firefox areas.

15. Add-ons Testing

  • Addon Memory Leak Testing is a big focus of the QA Memory Leak Testing Work
  • We test new Extensions uploaded to Addons.mozilla.org and also the Top-Downloaded and Recommend Addon's
  • Leak Testing is done with Debug Builds and also with Leak Gauge
  • We provide also a Leak Gauge How-To for Addon Developers and AMO-Editors
  • In the Future A successful Leak Test is a requirement for new Addons on Addons.mozilla.org
  • A best practices Document is in progress to give Addon Developers a hand in avoidingMemory Leak

16. Addon Testing Practice

  • We use new Profiles to avoid any false-positive Results from other Extensions
  • We test all aspects of the Extension (Install, Uninstall, Features of the Extension)
  • Provide Detailed Steps to reproduce so that this Memory Leak is reproducible by Developers and also to be able to verify the fix of this Leak.
  • In case a Memory Leak was found we file a Bug in the Mozilla Bug Database and inform the Extension Developer
  • Future Plan: We will use more Automation to do this testing !

17. Success so far

  • Successfully identified various types of Leaks in Extensions and Firefox Components also in specific Scenarios beyond automated Tests
  • Memory Leak Logs from Debug Builds and Leak Gauge Help Developers to identify the cause of the Leak
  • Memory Leak Testing of Extensions help to maintain the great Firefox 3 Performance

18. Questions ?

  • References:
  • http://wiki.mozilla.org/Performance:Leak_Tools Overview over Performance/Leak Tools
  • http://blog.pavlov.net/2008/03/11/firefox-3-memory-usage/Blog Post from Stuart Parmenter
  • http://blog.mozilla.com/tomcat/2008/03/21/extension-memory-leak-testing/Blog Post about the QA Extension Testing
  • Contact:
  • Clint Talbert (ctalbert on irc.mozilla.org)[email_address]
  • Carsten Book (tomcat on irc.mozilla.org)[email_address]