QA Lab ManagementMeeting new Business Challenges with Dynamic Quality Assurance

Mike AzocarSr. Developer Technical SpecialistMicrosoft Corporationmazocar@microsoft.com

A little about me…

So who are you?

Developer? Testers? Coachers/PMs? Analyst? Check signers?

“we have a changed requirement, what do I need to test?”

Have you heard any of these?

“my testers are spending too long testing the same thing”

“tooling is expensive (time, licenses, people)”

“developers and testers work in silos and don’t communicate/speak the same language”

“when is my software ready to ship?”

“the developers say the defects are useless”

Manager

68%

What if you could…

reduce the time it takes to determine the root cause of a bug

enable users to easily run acceptance tests and track their results

reduce the time it takes to verify the status of a reported bug fix

reduce the time it takes to replicate a bug uncovered by user actions

reduce the time it takes to isolate differences between the test and production environment

8

Welcome to the revolution

rele

ase

pla

nnin

g set test strategydefine done, doneestablish environmentsreview configurationscreate plansgenerate data

multiple iterations

What tester do

inception construction release

rele

ase

ite

rati

on regression

testingrelease doneness testingrelease sign off

itera

tion p

lannin

g add stories to plandefine acceptanceselect regression tests

itera

tion e

xecu

tion author tests

run testsfile bugsverify fixesautomate tests

itera

tion

retr

osp

ect

iveupdate master

planidentify product debtidentify test debtselect test for automation

Generalist SpecialistManual Testing

Some scripting

Creates scripts to set up lab, create data

Strong scripting skills

Some coding skills

Strong coding

Develops fully automated testing procedures

Expert coding skills

Where does testing happen?

Black Box Testing

White Box Testing

API Testing

70% of testing happens here

Majority of test tools target here

Visual Studio 2010 Test Capabilities

coded ui test

unit testingweb performance test

load test

test runner

test case management

lab management

data collectors (video, action log, event log etc)

team foundation server with reporting(bugs, test cases, requirements, user stories, source control, build)

generalist

specialist

12

Let’s drill down!

rele

ase

pla

nnin

g set test strategydefine done, doneestablish environmentsreview configurationscreate plansgenerate data

itera

tion p

lannin

g add stories to plandefine acceptanceselect regression tests

itera

tion e

xecu

tion author tests

run testsfile bugsverify fixesautomate tests

itera

tion

retr

osp

ect

iveupdate master

planidentify product debtidentify test debtselect test for automation re

lease

ite

rati

on regression

testingrelease doneness testingrelease sign off

What testers do

inception construction release

Virtual Machines

Hyper-V Host

Visual Studio 2010 Lab Management

Automated Build-

Deploy-Test Workflow

Reduces Hardware Utilization

Remote Diagnostic Collection

Virtual Environment Setup/Tear

down

• Built on Hyper-V• Integrated with TFS

2010• Enabled by SCVMM

• Integrated management• Multi-hypervisor

management• P2V and V2V• Template deployment• Optimize resource usage

• Managed by Microsoft Test Manager

Library Share 1

Stored VM 1

Stored VM 2

Stored VM 3

Build Controller

Test Controller

Visual Studio Lab Management 2010 Leveraging virtualization for ALM

Physical Environment / ESX

Build Agent

Test Agent

Machine 1

Machine 2

Machine 3

• Lab Management

• Build Management

• Test Case Management

• Project Management

• Source Control

• Work Item Tracking

• Reporting

TFS 2010

Visual Studio 2010Microsoft Test Manager

Hyper-V Host 1

Build Agent

Lab AgentTest

Agent

Virtual Machine 1

Virtual Machine 2

Virtual Machine 3

Windows Vista

SCVMMVM Template Library

Sysprepped “gold images”• Windows Server 2003• Windows Server 2008•…etc.• Stored in SCVMM Library

Templates and Virtual Machines

Windows 7

Windows Server 2008

Test AgentBuild

AgentLab Agent

Windows Server 2003

17

Let’s take a look:Managing Test Environment Complexity

Microsoft Confidential

Reporting on Testing Progress

Build progress overtime is tracked

Recent build status is reported to the

team

Reporting on Testing Progress

Are we getting enough coverage?

How many lines of code are we churning?

Reporting on Testing Progress

Workflow run for Deployment and

Lab Restore snapshots create new ones

after build

Test against Lab Environments Automatically

Failing Tests are Flagged for further

review

Amount of Code Covered by Test is

show

Amount of code checked in per

build

22

Wrap-Up

Microsoft Confidential

Your Creativity Unleashed

− Save time, focus on high value tasks− Fast Forward for Manual Testing− High quality bugs with single click

− Embrace Change− Rapid Setup/Teardown of Environments− Test prioritization to assess impact of code

check ins− Traceability and Visibility

− Rich Modern Experience− Elegant Task focused UI− Combined manual & fast forward for manual testing− Plan, Test, Track flow

Simplicity through Integration− TFS is THE collaboration hub

− Break silos between roles− Share Virtual environments between

team members

− Go Live Readiness− Performance Testing Early and Often− Highly scalable (Also 64 bit support) − Standard languages for scripts

− Quality Insight− Cross subject area reporting− Real time feedback

Quality code ensured− No more “no repro”

− Integrated, Extensible Diagnostics− Rich Bugs, fix first time, every time− Intelli-trace

− Automation to reduce Regression− Coded UI Test− Familiar language & toolset support

− Eliminate wasted cycles − Test in production like environments− Efficient, predictable build hand off

Call to Action

Download Visual Studio and Team Foundation Server RTM virtual machine for Windows Virtual PC and try Test Professional today! http://bit.ly/VSTFSVPC

Check out the Online MSDN Documentation – Testing http://bit.ly/TestApp

Questions and Answers

27

© 2009 Microsoft Corporation. All rights reserved. Microsoft, Visual Studio, the Visual Studio logo, and [list other trademarks referenced] are trademarks of the Microsoft group of companies.

 The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation.  Because Microsoft must respond

to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. 

MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED, OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.