Software Testing with Microsoft Test Manager 2012 and Lab Management 2012

Brian KellerPrincipal Technical Evangelist

USS Yorktown, SmartShip

• crew member entered 0 in a data entry field, caused a “divide by 0” error that shut down propulsion

• dead in the water for 2hrs 45mins

Ariane 5 Flight 501• re-used code from Ariane 4, but took a

different flight path because of different booster design

• conversation from 64bit float to 16bit signed int caused overflow (exception handler was disabled for perf reasons)

• cost: > $370 million @ 1996 prices

Vasa• sank after sailing less than 1 nautical

mile out of Stockholm

• specifications were faulty, failing to take into account enough ballast to keep the ship upright

• test teams revealed that it was unstable, but launch was rushed, and QA was ignored

Training

Conditioning

Climbing

Training Phase

Cost of Failure

Development

Test

Release

Software Phase

Cost of Bugs

Generalist Specialist

Manual Testing Some scripting

Creates scripts to set up lab, create data

Strong scripting skills

Some coding skills

Strong coding

Develops automated testing procedures

Expert coding skills

Tester Segmentation

Black Box Testing

White Box Testing

API Testing

70% of testing happens heremajority of test tools target here

What usually happens when a testerfinds a bug?

Graphic by Jeff Atwood (CodingHorror.com)

Unified ALM Solution

Team Foundation

Server

Planning SCM Work Item Tracking

Testing and Lab

Management

Build Automation

Reporting and

Analytics

demoMicrosoft Test Manager 2012

(1) Get source (2) Compile projects

(3) Copy build to running environment

(4) Run deployment scripts for each machine

(5) Create environment snapshot

(6) Executeautomated tests

(7) Send testresults

(8) Publish results toTeam Foundation Server

Build – Deploy – Test

Team Build

Team Foundation Server

Test Controller

VMM Agent Test Agent Test Agent

VM Host Web Server (VM)

Database Server (VM)

Virtual Environment

VMM Agent Test Agent Test Agent

VM Host Web Server (VM)

Database Server (VM)

SCVMM ENVIRONMENT

Test Architecture

Test Agent Test Agent

Web Server Database Server

STANDARD ENVIRONMENT

Test Agent Test Agent

VM Template 1

VM Template 2

VM TEMPLATE LIBRARY

Team Foundation Server

System Center Virtual Machine Manager

(SCVMM)

Team Build

Test Controller

SCVMM vs. Standard EnvironmentsSCVMM Environment Standard Environment

Build-Deploy-Test

Manual Testing

Snapshot / Restore

Share Bug Snapshots

Network Fencing

3rd Party Virtualization

Physical Machines

demoLab Management

Lord Kelvin, 1824-1907

“…you cannot manage what you cannot measure.”