- 1.
-
- OSCON 2008-Fixing Hard Problems Through Iterative QA and
Development
Clint Talbert - Carsten Book Mozilla Corporation 24.July 2008 2.
Introduction
- Memory Leak Debug Test Builds
3. The Approach
- Development and Test Development collaborate on tests and
tools
- Create widely applicable, reproducible tests
- Test Tools are treated as first rate products
- Implement a reporting structure for results
4. Collaboration
- Developers and Test developers bring different perspectives to
the table
- Developers concentrate on the deep tools to measure behavior of
code (i.e. Ref-Counting Analysis etc)
- Test Developers work on harnesses and reporting infrastructure,
lowering barrier of entry
- Having reliable test tools that are run and developed on a
regular basis help bring a sense of testing into the development
culture.
5. Reproducible
- Keep tests and tools simple
- By taking reliable test tools into impure systems, you can get
results that you can trust.
6. Test Tools Are Products
- Test Tools must grow with the product
- Test Tools must be simple and maintainable
- Test Tools must be developed and QA'd just like the product
they are testing
- Test Tools should be very easy to run
- Test Tools should require only the most basic level of
infrastructure possible to run. (But should also fit into a high
infrastructure scenario for reporting)
7. Reporting Results
- Use results to further a culture of testing
- Use results to understand if the test is reliable, and if so
broaden your testing scenarios
- Easy to understand results empower community members
8. Reporting Results
- What our results reporting looks like
9. Leak Testing Overview
- Lots of Memory Improvements for Firefox 3:
- The XPCOM cycle collector continuously cleans up unused memory.
Plus, hundreds of memory leaks are now remedied
- Memory Leak Tests after every Code change to make sure we
introduce no new Memory Leaks
- QA also started with a lot of Tests for Memory Leaks
- Mozilla QA runs manual tests using the Litmus Testcase Suite
(using Tools such as Debug Builds and Leak Gauge)
- Regression Memory Leak Testing of Firefox
- Memory Leak Tests of new and popular Addon's
- Future: More Automated Tests run by Mozilla QA
10. Tools
- Various Tools for Performance and Leak Testing
- QA uses Debug Builds and Leak Gauge
- Debug Builds: Debug Builds can be build with Trace-Malloc
Support to search for Memory Leak Builds every time up to date
build with latest checkins and overview over Leaking Components
(but Debug Builds need to be built individually and you need a
Development Environment (Xcode, Visual Studio Express, etc) for
most platforms)
- Leak Gauge: Developed by David Baron - It is designed to assist
in detecting what leaks of large object graphs occur during normal
browsing activity.The logging can be run during normal browsing
without significant overhead . Log taken by setting environment
variables in a release build.
11. Debug Builds
- Using Firefox Debug Builds with Trace Malloc enabled for all
Plattforms (Windows XP/Vista, Linux and Mac 10.4/10.5)
- Trace Malloc allows searching for Memory Leaks
- Can provide a Log File Output
- Identify Leaking Components, this information helps Developers
to debug and fix the Memory Leak !
12. Leak Gauge
- Easy to use Memory Leak Test Tool developed by David Baron
- Only Requirement is a environment variable and works on all
platformsc:NSPR_LOG_MODULES=DOMLeak:5,DocumentLeak:5,nsDocShellLeak:5,NodeInfoManagerLeak:5
- And set NSPR_L OG_FILE= c:leak1.log to d efine a Leak Log
- Works on Firefox 2 and Firefox 3+ Release Builds
- Leak Log Upload Form on
http://mxr.mozilla.org/mozilla/source/tools/footprint/leak-gauge.html
for analysis
- Upload Form indicates if a Memory Leak was found
13. Leak Gauge Log Output
- W hen a Memory Leak was found the output of the Log analysis
look like this :
Memory Leak 14. Litmus based Tests
- To make sure we have no hidden Memory Leaks in Firefox Mozilla
QA was running a FFT (Full Functional Tests) Testrun in Litmus(
http://litmus.mozilla.org )with Debug Builds on Linux, Mac and
Windows. Memory Leaks were Reported to the Bugzilla DB.
- Litmus is a Open Source Test Case Management Tool. Litmus
contains manual testcases and a Full Functional Test contains
around 750 manual testcases from all Firefox areas.
15. Add-ons Testing
- Addon Memory Leak Testing is a big focus of the QA Memory Leak
Testing Work
- We test new Extensions uploaded to Addons.mozilla.org and also
the Top-Downloaded and Recommend Addon's
- Leak Testing is done with Debug Builds and also with Leak
Gauge
- We provide also a Leak Gauge How-To for Addon Developers and
AMO-Editors
- In the Future A successful Leak Test is a requirement for new
Addons on Addons.mozilla.org
- A best practices Document is in progress to give Addon
Developers a hand in avoidingMemory Leak
16. Addon Testing Practice
- We use new Profiles to avoid any false-positive Results from
other Extensions
- We test all aspects of the Extension (Install, Uninstall,
Features of the Extension)
- Provide Detailed Steps to reproduce so that this Memory Leak is
reproducible by Developers and also to be able to verify the fix of
this Leak.
- In case a Memory Leak was found we file a Bug in the Mozilla
Bug Database and inform the Extension Developer
- Future Plan: We will use more Automation to do this testing
!
17. Success so far
- Successfully identified various types of Leaks in Extensions
and Firefox Components also in specific Scenarios beyond automated
Tests
- Memory Leak Logs from Debug Builds and Leak Gauge Help
Developers to identify the cause of the Leak
- Memory Leak Testing of Extensions help to maintain the great
Firefox 3 Performance
18. Questions ?
- http://wiki.mozilla.org/Performance:Leak_Tools Overview over
Performance/Leak Tools
- http://blog.pavlov.net/2008/03/11/firefox-3-memory-usage/Blog
Post from Stuart Parmenter
-
http://blog.mozilla.com/tomcat/2008/03/21/extension-memory-leak-testing/Blog
Post about the QA Extension Testing
- Clint Talbert (ctalbert on irc.mozilla.org)[email_address]
- Carsten Book (tomcat on irc.mozilla.org)[email_address]