Lessons from Survey System Testing at Microsoft: The Good, The Bad and The Ugly

Presented by L. Allen Slade

Microsoft People Researchat the

17th Annual Conference of the Society for Industrial and Organizational Psychology

Contact:L. Allen Slade

One Microsoft WayRedmond WA, 98052allens@microsoft.com

Lack of Systematic Testing: “The Bad” (1998) Actions

First global web based survey Vendor churn Trust and hope (i.e., don’t test)

Impact: Failure Survey locked up Vendor replaced 6 weeks of downtime Data integrity issues?

Lesson: Testing is necessary

Emerging Testing:“The Ugly” (1999) Actions

Very limited testing was planned New research team put together a very quick test plan:

Load testing Data integrity testing Full system testing

Testing results 25% decrease in capacity vs. previous year. Adding encryption

reduced capacity further Miscoding employee response.

Impact Good user experience. Fixes delayed survey launch by 3 weeks.

Lessons Testing must be planned. Always have slack time after the last test to fix your system.

Systematic Testing:“The Good” (2000) Actions

Selected new vendor Evaluated internal testing capabilities. Agreed to full testing as part of contract.

Partnered more effectively with internal IT. Did not test reporting ordering tool.

Testing results Identified load issues. Tweaked a few data issues.

Impact Fixes delayed survey launch 10 days. Virtually flawless experience for employees/managers. HR users: Report ordering tool had slow response time during

last two days. Lesson

Testing is a partnership. Always have slack time after the last test to fix your system. Test all components.

Systematic Testing v. 2.0: “The Better” (2001)

Actions/context: Refined the test plan – including load testing report tool. Outsourced tasks. Nimda virus hit day after survey launch

Testing results Outsourced testers not very effective No load issues identified. Test environment had only MS survey. Data integrity testing revealed bugs – in test data

Impact System not penetrated by Nimda virus. Downtime as precaution. Survey locked up on a few users. DLL conflicts between different surveys The “tilde” problem. Vision impaired employees unable to access survey.

Lessons Simplify where few bugs are identified Build test cases that approximate reality. E.g., 70% favorable responses Test survey in production IT environment Accessibility testing (or accommodation)

The Impact ofSurvey System Testing

1998The Bad

1999The Ugly

2000The Good

2001The Better

Response 57.7% (15,824/ 27,444)

66.4% (21,323/ 32,089)

80.8% (32,123/ 39,753)

86.4%(41,252/ 47,721)

Downtime 6 weeks 10 hours 2 hours 24 hours

Report Ordering

Few reports 1,550 reports ordered offline through Excel

2,748 reports ordered online

2,226 reports ordered online4,774 reports pushed thru manager portal

There Never is A Final Version

A flawless survey would still need to be tested the next year: Technology changes Systems are complex